Computing Reality

Isn't Gerrit Ironic?

noreply@blogger.com (Miles Parker) — Sat, 05 Jan 2013 00:26:00 +0000

In my last post, I described the current state of Eclipse Gerrit adoption. If you haven't read that post, you might be as surprised as I was to find out how low it is: less than 15% of all Eclipse Projects are using Gerrit, and of those only a few are using it in a meaningful way.

Implementing Gerrit

That's understandable, because Gerrit is notoriously difficult to implement. But for those intrepid Eclipse Project Leads who don't mind getting their hands dirty, I'll take the time to detail all of the laborious steps:

Click Here Now!!
Replace the summary text "my project" with your project name
Click "Submit Bug"

If you're an Eclipse Committer or Contributor, the steps are even simpler.

Email a link to the instructions above to the project lead.
(There isn't a step two.)

Of course, once you've enabled Gerrit for your project, there are all kinds of new processes and practices that you'll need to adopt:

(There aren't any.)

Actually, while we're on the subject of code reviews, I thought it might be appropriate to try a little exercise. Please compare and contrast the following words:

Facetiousness
Sarcasm
Irony
Satire

Question: Did you feel the need to google any of these terms before answering? If so, you might be particularly interested in this link. I think the most important thing to learn there is don't debate irony.

On reflection, it turns out that the above exercise wasn't at all appropriate or relevant. I was just wasting your valuable time with an (arguably) mildly humorous segue. So let's just move on and forget it ever happened, shall we?

But before we do that, one more question: Which of the above words is most appropriate for describing this statement? (You may also choose "5. All of the above", "6. None of the above" or "7. ~~Other:~~ Just Stupid".)

Okay, enough [1, 2, 3, 4, Other]. Everyone knows that code reviews are serious business. Deadly serious business. Lives are at stake. Remember that the next time you blindly "Merge", Mr. or Ms. Reckless Coder!

Why Gerrit?

Really, the better question is "Why Not Gerrit?" -- or it would be, except that it would be really odd to introduce a query by proposing a negation. There really is nothing to lose*, except for poor traceability, error-filled code, and the ability to easily slip one past the Eclipse IP team. [Irony, I think. Or was that Satire?] Gerrit adoption can be as steep or as gradual as you want it to be -- I believe all of the default settings allow current committers to continue to commit straight to the Git repository, but I'm too lazy to check right now.

*There is one important exception; users of the Indolent Development methodology have found significant incompatibilities with Gerrit, as well as code reviews in general. It can especially interfere with practices 3 (The Random Walk) and 9 (Self-Expression). But the vast majority of Indolent adopters would have been too lazy/smart to read this far anyway.

Advantages

The advantages to code review are manifest (translation: "everyone claims this, but I'm not sure if there is any hard evidence, and am too lazy to find out"), but include:

Dramatic code quality improvement.
Enforcement of common idioms and project standards.
Clear workflows and defect traceability for contributions.
Encouragement of intra-project collaboration and knowledge sharing.
Greater inter-project transparency.
Promotion of a more open and equitable Eclipse development process. (Back to that in a second.)
A bunch of other stuff that, again, I'm too lazy to look up references for. But why am I always the one who has to do all of the work? If you're so damn curious, why don't you just Google "Gerrit Advantages" or something?

Disadvantages

Okay, there are just a few disadvantages. Actually, so far I've compiled a list of 27 (I'm actually not joking), but I'll cover those in my next post. The reason I'm not sharing those now is that I really don't want to give you any information that might dissuade you from Clicking Here Now!! Sound fair?

An Eclipse Revolution?

The reason that I don't want to dissuade you from adopting Gerrit is because of point 6. Gerrit has the potential to profoundly democratize the contribution process for Eclipse. That's because -- when implemented fully and correctly -- Gerrit makes it every bit as annoying for Committers to get code in as it is for Contributors.

Hmmm...let's see if I can put a more positive spin on that....

For current committers -- even when fully implemented -- Gerrit really doesn't add much more of a burden than a straight commit, assuming that your project has decent oversight and communication. (You have that already, right?) And in many ways, it actually makes that process easier by automating some bits. The only additional step is that you have to actually go to the Gerrit review and click the submit button. At that point, you might say to yourself, "gee, maybe I should ask at least one other person what they think of this change before I inflict it on all of my consumers and users" -- but that's strictly between you and your conscience.

But for contributors*, the whole picture changes dramatically:

Before Gerrit

The traditional process involves creating (or taking on) a bugzilla bug, submitting patches (the process of creating an Eclipse patch is far from trivial), hoping someone would bother to take a look at the patch and make comments on it, digging through code trying to find the bits that those half-thoughout comments were referring to, merging and making changes to that patch, dealing with some rather opaque IP rituals, etc..

For the committer handling the patch, it mean manually applying the patch, dealing with any dependency changes or other build oddities, seeing if the darn thing actually builds, playing around with it, making half-thought out comments -- all of the time secretly hoping that the contributor would just give up and leave you in peace [Satire].

Git helps quite a bit, but it really doesn't help with the most important bits; managing builds, handling workflow, and tracking concerns.

After Gerrit

Now, if you want to contribute code to a Gerrit enabled project, things are much simpler. You create a bug, create a local topic branch, commit push them to gerrit, add project reviewers, and as changes come in, amend your local topic and push it again.

For the committer, it's even simpler. You fetch the commit (plug: using the rapidly improving Eclipse Mylyn Gerrit connector!), run it, suggest changes, wait for the user to get everything perfect, and then click one little submit button -- actually grateful that someone else fixed a bug or made an enhancement that you won't have to worry about anymore.

I really think that that advantage outweighs any other concerns, and it's why I'd really ask Project Leads to Click This Link Now!! Remember, you have nothing to lose!! Web-masters are standing by!! Believe it or not, that kind of badgering, repetition, naked-appeal-to-ego and strategic use of colour and obnoxious font styles has been repeatedly shown to work, even for very intelligent people like you. So Click Here Now!! (Actually, please don't do that unless you actually are a project committer.)

*For those of you who are not familiar with the Eclipse community, a "committer" is someone who is a formal member of a project and has been given commit rights to the repository, a "contributor" is all of the rest of us schmucks. The 99%'ers of the Eclipse universe, as it were.

Adopting Gerrit

Okay, things are a little more complex than I've made it sound, admittedly. But it really isn't that hard to get going on. Check out this wiki page, but don't let the details scare you.

Since Gerrit isn't actually all that easy to use properly, sometimes for purely social reasons but also because of purely technical ones, -- especially when it comes to reviews that depend on other reviews (run away! run away! while you still can!!) -- and since new users will need quite a bit of time to get used to it and to become familiar with all of the conventions, it might make sense to start out with simple workflows on relatively straightforward bugs just to get the hang of things and to allow your project process to evolve. But you can get a good survey of best practices by checking out EGit and Mylyn reviews as both projects have been doing this long enough to have evolved pretty mature processes. (Hint, poke around and find all of the comments from current committers pointing out all of the dumb things that new committers are doing wrong. Here's a good one.)

For project leads, you'll need to decide how err.. persuasive you want to be about getting people to adopt Gerrit. There may be some resistance from current committers, so there might be a need to get a little more active in encouraging adoption. Once you get a few committers rolling, my guess it that the inherent advantages along with the social/network pressures will kick in. Especially when existing committers start to see those contributions come pouring in.

At this point, I should probably make some suggestions about best practices, offer some tips and tricks and address some common pitfalls. But really, I'm feeling too lazy. Hey, it's Friday, okay? And I'm just back from holiday. I haven't really sunk into the demands of real work yet. Anyway, what are you doing reading this blog? Don't you have more important things to do?

Like...

If you're an Eclipse Project Lead, you've already won!!! Click This Link Now!!

And for all of you potential Eclipse Contributors out there, Mylyn, EGit, CDT, JDT/Platform and a bunch of other projects are already open for business!

On Computing Reality

noreply@blogger.com (Miles Parker) — Thu, 03 Jan 2013 20:45:00 +0000

This blog just passed a milestone, with 100,000 page views. Probably not a lot compared to most technology blogs, but it's gratifying to know that people have found the blog worthwhile enough to return to. Hard to believe, but I've written 75 posts in the last 3 years and 9 months. It's easy for me to remember how long this blog has been going, because I started it just a few days after my second child Ziji was born.

Having reached that milestone, I think it's time for a bit of a shift. Over the winter holidays, I've been thinking a lot about how to balance work, life and the more creative things I feel compelled to do. One of those things is writing. In the last year, I've only written 3 posts. My New Year's resolution is to blog more, but I've felt that if I'm going to take the time away from all of the other things I need and want to do, I need to become a lot more disciplined and focussed when I do take that time. And that means better defining what I'd like this blog to be about so that I can decide what's worth writing about and you can decide whether it might be worth reading. So I've done a little "rebranding" to use bloodless marketing speak and I thought it would be worthwhile to take a little time to explain why.

What's it all About?

Why "Computing Reality"?

My blog's new title invokes what to me is the key paradox in software development and science: We can represent interesting aspects of the world we live in using computational methods, and those representations can give us profound insights, challenging our biases and preconceptions. And I feel strongly that algorithmic approaches such as agent-based modeling are far better suited to describing reality than equation-based approaches. But we make a dangerous assumption when we accept the now trendy view (Kurzweil, et.al.) that we can really capture (in every sense of that word) reality in a stream of binary numbers. To me, that's not a exciting futurist concept but just the inevitable, logical, dark, and just plain wrong-headed endpoint of 400 years of scientific materialism, which is really nothing more than the state religion of the Silicon Age.

Reality is not computable. That shouldn't be a controversial statement -- it seems to follow naturally from the work of the greatest minds of the 20th Century, including Gödel, Turing, and Feyerabend, as well as a bunch of annoying post-modernists. But that doesn't imply the magical hand of an external being either. Nor does it imply a meaningless existence. Reality is far more mysterious and beautiful than either of those extremes: We can never describe reality, we can only experience it, and that experience unfolds in ways that we'll never be able to predict.

Along that line, the blog's name reflects Contemplating Reality, a brilliant book that -- while ostensibly for Buddhist practitioners -- could be a great resource for anyone who is interested in rigorously analyzing their own internal cognitive biases and in challenging their own assumptions about what reality is. The book describes simple practices -- we call them analytical meditation, but that's just another label -- that you can use to discover the fundamental nature of reality that exists beyond concepts. An astute reader will ask at this point "but then how can you contemplate it!?" The answer is that concepts can be useful without them having to be real!

In his article Reality Isn't What You Think, Andy Karr has to say about this apparent paradox:

"Genuine freedom from fabrications comes from seeing through the delusion of concepts, not from suppressing them."

Here's an example of how analytical meditation works:

"You might feel that you know where various geographical reference points are located—Asia, Canada, Kansas, San Francisco. With your mind’s eye, see if you can find them. Are they in the east, west, north, or south? Did you take into consideration the curvature of the earth? How far away are they? Look at your experience of each reference point in turn. Is “Asia” anything more than the memory of a shape on a map? Can you ever find a whole city, or is San Francisco made up of lots of different glimpses the way home is? By contemplating in this way, you will understand that the planet and its geography are also projections. All outer objects can be investigated in this way, and you will find that all are emptiness. Outer objects still appear, but they have no essence."

We can approach any material or conceptual object in this way, teasing it apart and discovering the empty quality inside, and by doing so we can open our minds to a much greater set of possibilities. One of the things that I'd like to do in future posts is to apply this same kind of practical rigour to challenge our assumptions about common representational problems in software and science.

For example, we might ask ourselves what a "Class" really is: Is it possible to create a class that actually represents a domain concern completely? If not, what aspects are arbitrary? Is there still a consistent way to represent these arbitrary aspects? And so on. I discuss the scientific implications of this approach in this video.

These are the kinds of questions that I ask myself all of the time as a software developer -- and I suspect that I'm not the only person who looks at problems of representation using a similar sort of deconstructive approach. Basically, rather than look at the donut, we look at the hole. I find that activity to be an enormously helpful -- and enjoyable! -- part of the process of software creation. Because the point of these kind of exercises is not to prove that the concepts are useless, but to see for ourselves how they constrain and condition our designs -- from that perspective we can start to play with software abstractions rather than worshipping them -- gaining a greater appreciation for the limits and possibilities of all design decisions.

When we take this contemplative, reflective approach, we are simply applying the same scientific approach to our mental projections that we apply to matter. There is nothing "religious" about that -- just a willingness to be completely open to the further questions that arise when we do so. I think that anyone who does serious investigation into the matter would also end with the conclusion that Reality cannot be conceptualized. And somehow, this leads us naturally back to bigger questions, like "Why does it feel like something is missing?" or "Why do people sometimes treat other people so badly?"

If such talk makes you a bit uncomfortable, I can assure you that I feel just as uncomfortable mentioning it. People aren't comfortable discussing "religion" and given the history of repression and ignorance brought about by it it's not difficult to see why! Especially since smart, creative and iconoclastic people have usually been the targets of that repression and ignorance. When people do decry the secularization of society it's usually because it is their version of the truth that's getting replaced. For what it's worth, I have absolutely no interest in convincing people that my particular approach to truth is the correct one. But I find myself swallowing my tongue when it comes to discussions that revolve around what I've come to think of as "Computing Reality", and that doesn't feel right either.

Because pretty much everything that happens in my life is framed by the lens of my personal practice and faith tradition, which happens to be the Buddhadharma -- and if I'm going to take the time to write about what I want to write about, it's going to include that, even if I keep the explicit references to a minimum. To many people, especially us rational technology types, Buddhism seems to be a relatively non-threatening but still fuzzy headed religion that teaches people to be nice to each other, and is maybe a bit too hip. But to me, it isn't an "ism" at all -- it's simply a very honest, rigorous and direct approach to exploring one's own experience. And my experience is that the world is a pretty magical place.

Ultimately, there is a problem with trying to separate the analytical view from a more sacred vision: when we try to separate spiritual issues from the "real-world", we're biasing every single decision we make away from the really deep, basic concerns: What makes people happy? What causes suffering? We don't have to talk about them all of the time, but we should at least care about the answers to these questions and resist the temptation to shy away from them.

Finding Balance

So, returning to the question of balancing work, home, and blogging. There are two major competing interests here:

First, I made a real commitment to not work after hours and to spend as much time as I could with my family. As an independent entrepreneur, the workday never really ended, but now that I have a real job I've been able to return some basic sanity to my life. I can be more disciplined setting limits, and being able to do that has totally transformed my family life.

Second, it's also really hard to justify spending time writing blogs during work hours. We all recognize that blogs can be a great tool to help companies communicate, but that's a pretty soft value, and difficult to offset against immediate actual income producing tasks like delivering on contracts or putting the pieces in place to get that next sale. Tasktop is still a young company, and that means that every choice we make involves a significant tradeoff. To put it in more stark terms, it's awfully hard to justify spending the time to write a blog when your colleagues have just spent a good part of the weekend solving a critical customer issue -- time that they are away from their families.

Finally, there is the issue of what I should write. How can I find a better balance between a) what I know people will actually want to read about: "how can I apply technology X to solve my problem" -- and b) what I what I want to write about: "what are we doing here, and what if anything does it all mean?" My blog appears on a number of tech feeds, and I don't want to abuse that privilege, especially since those questions get into social, philosophical, political and (the scariest word of all) spiritual issues. At the same time, I find myself doing a lot of self-censoring, and I'm feeling more and more uncomfortable about that.

I'd love to hear what other frequent bloggers and blog readers have dealt with making these kind of balances. Please share your experiences by commenting here, or feel free to contact me directly (email milesparker at the email-provider-everyone-else-uses) and I'll include them in a followup post.

This is what I've decided: When it come to pure technology application, I'm going to carve out a bit of work time to write more focussed articles on issues directly relevant to what we're doing at Tasktop, but I'm going to make sure the human dimension doesn't get left out.

And I'm also going to take some time at home to write more about the more esoteric -- for lack of a better word -- issues that I want to explore, still rooted in a technical foundation that is relevant as possible to my existing readers but hopefully also inviting to a less technical audience. While I respect that everyone won't want to hear about my personal experiences and interests, part of finding a balance for me is to bring more of these bigger questions into this blog. That, and yes, model trains, and other things. Real communities don't make arbitrary divisions between "work" and "life", and I don't think we have any business building tools without having an appreciation for the absolute limits of our tools contrasted to the unlimited potential we have as human beings.

Like it or not, as technologists, we're the priests and priestesses of the Silicon Age. What we do has far more impact on the lives of others than I think most of us appreciate. Many important social, cultural and political innovations have come directly out of the technical community. It's no accident that the Open Source movement has given rise to some of the most profound cultural changes in the last twenty years. I think it's time we recognize our changed role in society, and yes, as corny as it sounds, use all of our intelligence, creativity and goodness to make the world a better place for everyone.

A month after I started this blog, I read this poignant post from Ed Merks, on the loss of his partner of 27 years. No-one who sees the photo of Larry or who reads and contemplates Ed's poem should have the slightest doubt that all of life's experiences were meant to be shared. I have to say that reading Ed's posts was the moment that I felt truly connected to the Eclipse community and to the broader community of technologists out there. So thank you to everyone for being so open, in every sense. I'll try to return the favour.

Analyzing Gerrit: Java, Knime and Numbers

noreply@blogger.com (Miles Parker) — Wed, 02 Jan 2013 19:27:00 +0000

How much has Gerrit changed software development? ~~Well, consider that a couple of years ago most of us had never heard of Gerrit, and now it seems that you can’t contribute a line of code without it.~~ That’s how I planned to start this post, but it didn’t turn out that way. I should have said "and now Miles Parker can't contribute a line of code without it". Maybe I should take that as a sign of something...let's just say that they invented "deny fast-forward" for people like me.

So originally, this blog was going to be about how Gerrit was eating the world -- how it had come to dominate all software development at Eclipse. Because that’s what I actually thought, even though I had no real evidence to back up that assumption. In fact, it turns out that my perception of what was happening in the world at large was completely driven by what has been happening in my little corner of it. The surprising thing about that is that no matter how often that happens to me, it still surprises me that I can hold that much of a cognitive bias, even when I should know better!

Because as I started to look at the global reality rather than my own local knowledge, it quickly became obvious that the use of Gerrit at Eclipse is actually a lot less significant than I had thought. I had made the classic error of extrapolating an entire curve from two data points. Worse, I had failed to consider that my sample was clearly self-selecting, given that my company makes tools that integrate and converge Application Lifecycle Management (ALM) tools and yep, Gerrit happens to be one of those!

At Tasktop and within Mylyn, Gerrit is indeed triumphant. It’s amazing how quickly practices change -- when I arrived at Tasktop, we weren’t really using Gerrit. Now, while it is still technically possible to commit code to our Tasktop internal repositories directly, in practice it very rarely happens. Everything we do flows through Gerrit. The two projects I have been most familiar with over the last six months at Eclipse so are Mylyn and EGit. I thought they were the norm, but they turn out to be outliers. That would have been obvious, had I just turned to this page. Compare that to the number of git repositories at Eclipse. (And now that the CVS/SVN transition is complete, that includes everyone.) Counting all of the Mylyn projects separately, that adds up to coverage over 38 projects out of 240 giving an Eclipse Gerrit adoption rate of 15%. There are 31 “enable gerrit” bugs listed here. (Hopefully that number will be much larger if you happen on this post six months from now.)

But what I was interested in was how the uptake had occurred across projects. I thought it would be neat to begin to get a handle on the social dynamics involved in adopting a disruptive and disrupting technology like Gerrit. And so I got diverted for the next few days by the siren call of data analysis. If you're interested in the details, see below. But here are the pretty pictures:

First, it shouldn't be surprising that EGit has been running Gerrit form the start -- still, it's encouraging to see that they are walking the walk. In fact, from my experience working with people contributing to the project, EGit has a very deep Gerrit culture.

EGit

CDT has had some significant adoption, but they still seem to be in the testing-the-waters phase of adoption.

CDT

Interestingly, both core projects on Gerrit have yet to really adopt Gerrit in anger -- the usage so far is exploratory at best. One of the really interesting things here is that the people involved in these reviews seem to be coming from the EGit community -- a great example of the social coding dynamic in progress.

I should point out that even if projects don't adopt Gerrit wholesale, it is really great to see them support it -- if nothing else because it gives contributors a much better way to get their code into core feature projects. I'll discuss that issue in more depth in a future post.

JDT Core

Platform Resources and Team

And last to Mylyn, which has the most interesting and dramatic uptake.

Mylyn (All)

Note that the large spike in non-Gerrit changes in October 2012 is an aberration. That's due to a reformatting of the entire R4E code base -- while I happened to be in the middle of a review, necessitating the "rebase from hell" -- but that's a story for another time, perhaps with a cocktail in hand...

If we look at the overall ratio of commits, the picture is even more dramatic:

Mylyn (All) Gerrit Usage

That's going from 0% Gerrit usage to 75% in a little over a year's time. I'm biased, but I think that's pretty cool. It's also a perfect example of the kind of thing that Eclipse's strong emphasis on meritocracy and strong project leadership enables. Leadership -- in this case, Steffen Pingel -- is critical in driving these kind of deep changes in practice, because they hold the supply of carrots and sticks necessary to make people adopt it. Developers like to do things the way they've always done them before -- and so while sometimes we adopt these changes happily, other times we have to be dragged in kicking and screaming. Because it isn't always an easy transition to make, and it's not without cost -- an issue I'll cover in an upcoming post. The bottom-line is that it would be hard to imagine an open source project making this kind of transition organically.

The Tasktop transition has been even more dramatic, but for obvious reasons I can't show you the chart for that. One advantage of internal development over Open Source is that you can simply legislate adoption of new best practices -- you don't have to wait for the community to decide to go along with it!

Methodology

Since you (didn't) ask, here's how I made the pretty picture. By the way, I really like the word "methodology". It sounds a lot more scientific than "how I did this".

Since there is no simple way to say "give me all of the git projects that use Gerrit", I took the following approach:

Collected all commits to master from the git log. This ensures that we only count those Gerrit reviews that actually make it in.
Counted any commit that had "Change-Id:" in the message as a "Gerrit Commit".

Collecting the Git Log

Because I was too lazy to learn bash and too snobbish to use Perl, I spent far too much time creating a git log reader/translator in Java. It turned out to be the kind of thing that you-would-never-do-had-you-known-before-hand-how-long-it-was-going-to-take. But now that I've done it, perhaps it will be helpful to someone else. You can find all of the source code along with the data I collected in my GerritAnalysisTools GitHub project.

The relevant code is here. All I'm really doing is executing the git command themselves, and then parsing the resulting unhelpfully verbose git log data into a more structured format.

Execute it like this:

Main class:

name.milesparker.gerrit.analysis.CollectGit

Arguments:

"{Base Git Directory}" "{Data Output Directory}" "{git project names, comma seperated}"

Analyzing the Data

I used Knime for this next part. Except for the fact that they use GPL, and I have some reservations about that, it is a really neat Eclipse-based project. If you have any interest in Data Analysis (call it "Big Data" if you want to sound hip), you really should check it out. Knime also demonstrates that Visual Languages can work, if the domain it appropriate and reasonably specific.

This is what my workflow ended up looking like:

Here are the actual steps, going in row order:

Read in the file, renaming and cleaning up the columns in the process. (Looking back, I could have done a lot of the preliminary parsing work that I did in Java here, but that probably would have been significantly slower and pointless, given that most of the java side stuff was regexp manipulation.)
Coerced the strings into time objects, and pulled out the Year and Month into a single Year/Month column for binning purposes. (See steps 4.2 and 4.3 below.)
Added up the values for Insertions and Deletions into a single Changes column.
Manipulated the data. This is the interesting part, and the kind of thing that Knime kicks ass at. If you've ever gotten frustrated playing with pivot tables and groups in Excel and wondered why someone couldn't do a better job at this -- someone has.

Calculated the domains for dates and the Gerrit flag. Calculating domains basically creates a full enumeration of all possible values in a given column, allowing Knime to handle them efficiently in down stream nodes.
Grouped the data by Year/Month and the Gerrit flag. This collapses each set of rows with a unique combination of those two values into a single row applying whatever aggregation measures you want to use. In this case, I simply summed everything up.
Pivoted out the Gerrit flag. Form the previous step, each unique Year/Month has two rows -- one for each possible Gerrit value. The pivot takes the values in the two rows and creates new columns for each of these possible Gerrit values. (This is the kind of thing that is much more difficult to explain than it is to see for yourself, so I've included captures of the before and after below.)

Cleaned up the column names.
Built Interactive tables, histograms, and scatter plots.
Those are cool, but they aren't terribly attractive. So I wrote everything out to a CSV file, which I use in the Visualizaiton section below.

Step 3

Step 4.2

Step 4.3

Step 6

Presenting the Data

Since I care about what the Data looks like, and the Knime output looks like it was drawn on an Etch A Sketch, I took one more step, and exported the data into an Apple Numbers spreadsheet. I played around until I got things looking okay -- but you can judge that for yourself, because it's what I ended up with above. The good thing about Numbers is that it produces nice looking charts, and has a typically charming and simple Apple User Experience. The bad thing about Numbers is that it is a horrible application for getting any real data work done. I guess it's fine for displaying your company's astounding next quarterly profit return -- thus making great demos for Apple press events -- but it chokes when presented with any significant workload. By chokes, I means "freezes and crashes" -- not really acceptable behaviour for any software made in 2012.

Denouement

I forced myself to stop playing around with Numbers -- so that I could move on to something more productive, like actually writing this blog entry. The one that was supposed to take just a few hours, and ended up taking 2 1/2 days. But the nice thing is that I now have a repeatable process for creating pretty pictures of stuff that people could probably get a pretty good idea about anyway just by doing a few basic searches and looking at the text results. But hey, isn't that what Data Analytics is all about?

Virgo Tooling M4

noreply@blogger.com (Miles Parker) — Wed, 09 May 2012 18:53:00 +0000

I'm happy to announce that the Virgo Tooling M4 release is available. If you just want to find out more about that, skip ahead to the section below... But I realized that it's been a long time since I've posted and thought I'd use this opportunity to update folks on what I've been up to and give some background about the Virgo Tooling effort.

I've been really focussed on Tasktop projects during the day and haven't had much time to surface out into the broader open source development world. I'm also using the excuse of having a real job to try to have a life -- like spending time with our 1 1/2, 3 and 10 year olds and giving mom a break every once in a while, having time to reengage with meditation practice, and even spending a couple of hours a week getting our model train layout built. Basically, trying to ween myself away from the 24-hour a day developer mentality that seems to creep in when you're working on your own. One of the things that I've really appreciated about being at Tasktop is the family focus here; generally people actually go home at a sane hour of the day -- all too rare in the startup world. So I've been working at calling it a day around six (OK, sometimes I don't stick to that) and I've committed myself to no more late night blogging sessions! Still, I'm going to try to squeeze in some more blogs now that I'm settled in a bit. And this one happens to have an official purpose -- I want to share where we are with the Virgo Tooling effort.

Interestingly, one of the first things I started working on after joining Tasktop was Virgo Tooling; a perhaps less known aspect of Tasktop is that we partner with other companies to support Eclipse-related tools development efforts that aren't neccesarily Task related. (I have to say that that probably has something to do with the excellent reputation Tasktop engineering has in the Eclipse community. If you're interested in joining us, I should mention that we're hiring.) And we have a great relationship with the folks at SpringSource, now part of VMware. Leo Dos Santos, Ducky Sherwood and others have contributed a lot to the STS and other tools.

Now, I've never really been a web-centric developer, so I hadn't been exposed to all of the great tools that SpringSource has provided over the years. VMware and SpringSource have very genorously donated big chunks of these tools to the Eclipse project. We're helping to take the foundation provided by the Spring Source tools and integrating them with the entire stack of Eclipse runtime, web and plugin development tools.

But Virgo is definitely not just for web developers. In a nutshell, it allows users to run OSGi bundles easily from web services (but it does a lot more than that). What that means for Eclipse developers is that we now have a way to easily deliver all of the features our plugins provide outside of the Eclipse UI box. Because of that capability, Virgo is the one part of the stack that I did have prior experience with. I spent a bit of time working with the Virgo runtime when I wanted to deploy some EMF editors as RAP applications. I was having a hell of a time getting this to work on a bare Tomcat server, and Virgo made all of that easy. As I said I said in this post last year, "I haven't been that impressed by a technology that "just worked" since the first time I built an EMF model and editor".

A lot of the work that Leo and I started out on was a bit mundane, really -- sometimes there are things you just need to do in order to move ahead. One of the biggest things we've taken care of is excising all of the pesky dependencies from Virgo to the originating Spring Source tooling. That turned out to be a really big job, and it was kind of interesting work hunting all of these pieces down and finding replacements for them or in many cases, removing redundant or non-relvant code altogether. There is something rewarding about making code smaller and more modular. Naturally, we also worked a lot on updating features, branding, etc.. -- all of the necessary overhead that comes along with being an Eclipse project.

But after tackling all of that, we've had some time to make some improvements that you can actually see. Here are the biggest changes that you'll see in the latest version. (By the way, wondering what happened to M3? There wasn't one -- we're synching our milestones up to the Virgo Runtime releases, so we skipped it.)

What's new in M4

Improved Server Support

Virgo now supports almost all versions and types of Virgo installs, including older versions of Virgo and Tomcat, Jetty and Kernel installations. Best of all, you can now define servers using a single Server Runtime and the tooling will automatically discover the appropriate server configurations. You can even change server installations and versions for server runtimes without having to maintain any server settings.

UI

We've provided a number of new features aimed at improving the user experience. We've been giving more transparency and leverage into what's actually happening on the server. Right now, interactions with the server involve frequent shifts back and forth between the Eclipse IDE, the native file system and the command line. Our ultimate goal is to have tools powerful and complete enough that you can work with the server without ever having to move out of Eclipse.

Virgo Perspective

You can now get your own perspective on Virgo. We've organized it to support runtime exploration but it should be usable for general development as well. Please give us feedback on the overall setup; we'll incorporate the feedback in future releases.

Views

When we talked to current users of the Spring Source / Virgo tools, one of the things we discovered is that there were a lot of features that they wanted to have us implement that actually already existed, but in many cases were buried somewhere -- in many cases on a page in the server editor. So we're providing some of the information and tools that are provided in the WTP-based Server Editor as stand-alone views and enriching it. The idea is provide better insight into what's happening on the server side. Some of these features involve significant changes to how server resources are treated and we'd appreciate your feedback on how well they work for your usage scenarios. We'll use these changes as a basis for further enhancements.

Outline View

As a first take at helping users find what they're looking for, we provided an outline view that gives quick access to server artifacts.

Artifacts View

A new view supports transparent access to runtime artifacts. Currently, you can view bundle jars and libraries, but we'd like to support plans, pars, properties and other artifact types as well. But the real news is what we've built behind this -- Virgo runtimes are now exposed as Eclipse projects, giving you full access to the contents of jars, including classes. You can even use JDT tools like the Type view on Server side jar contents.

All of the new views are supported through the Eclipse Common Navigator Framework (CNF), which means that the views are highly configurable, and we've provided buttons to allow you to easily toggle on and off artifact types and present the artifacts in a tree or list view so that you can see all installed jars in one place.

Servers View

And because the views are all supported through CNF, you can easily customize them. For example, currently you can access all of the bundle information from the server view -- but we'll probably disable that by default in the release. In any case, you can set it up anyway you want by adding or removing the content provider. Again please let us know if you think of a way that we can improve the user experience for you.

Documentation and Help

The complete virgo documentation set -- including the Tooling, Programmer and User's Guide as well as the Wiki pages -- is now included as Eclipse Help.

Installation Process

Virgo Tools can now be installed from a single update site location.

Quality

Extensive bug fixes and re-factorings.
All Remaining Spring Dependencies have been removed.
Updated to meet Eclipse branding, packaging build and testing standards.
And yeah, there are still a lot of open bugs.

Check it out

So what are you waiting for? Get the milestone at http://download.eclipse.org/virgo/milestone/tooling. You can download a Runtime to go with it here: http://www.eclipse.org/virgo/download/milestones.php. (Hey, auto-provisioning of a runtime -- now that would be a cool little feature..)

Joining the Tasktop Team

noreply@blogger.com (Miles Parker) — Tue, 10 Jan 2012 02:06:00 +0000

Last week I said that I had some really exciting news to share, and this is it: As of today, I am part of the Tasktop team. I've long been an admirer of Tasktop, for a number of reasons...

First, as all Eclipse users know already, they've built some really cool Eclipse technology including the Mylyn task-focussed interface. But many companies have built cool open source tools...

More impressive is to take those cool ideas, discipline them and build a growing, dynamic company on top of them. But plenty of companies have also done that...

But what's really impressed me about Tasktop -- what's made me want to join rather than observe the effort -- is that Tasktop has gone far beyond the "packaged solution with bundled service and support" that typifies most open source derived business efforts. Many companies take a "fill-in-the-blank" approach to open source business models, creating a captive open source market around their offering. Tasktop has created and nurtured a healthy open source eco-system around core open source offerings, but it then re-imagines and re-purposes the technology, leveraging that technology and expertise to uniquely address real customer pain. That takes real vision, and to me it's a clear signal that the Tasktop leadership is able to imagine and execute at an entirely different energy level.

So I first approached Mik and David in the middle of last year to discuss what they were doing and how I might add real value to the effort. That lead into consulting work with Tasktop last year developing the initial implementation of what has become the Model Focusing Tools project. That was a great opportunity to get to know some of the team and the Tasktop way. Everything I saw then fit nicely with what I'd already intuited. We have a really great combination of engineering excellence, creativity, and light-weight organization.

It's nice to say "we" again after over seven years of working by myself or as part of a small team. Yes, I had a lot of close associations with very smart and capable folks through collaborative science efforts and open source projects, but it just isn't the same thing if you're not working toward a shared set of goals. And I hadn't realized just how much I'd missed having colleagues to work with on hard problems together until I got involved in that last project.

This morning as I've checked in my email, I've been getting emails from everyone at Tasktop welcoming me to the team. I must admit to a bit of cynicism about the whole "team" thing -- like so much else, it often seems an empty set of words that don't match up at all to reality -- but in this case it feels very genuine. So heartfelt thanks to everyone -- I'm working remotely so I won't get to meet you in person right away, but I look forward to it. And don't worry, you won't have to "buy me lunch". ;)

(And yes, I'm afraid to say that you'll still be seeing irreverent and irrelevant blog entries on Eclipse Planet, but I am going to try to keep a closer eye on the four-letter word count.)

Obviously I'm going to be focussed on and very occupied with my new role at Tasktop, and you may be wondering how this will affect the other efforts I'm involved in... With the new Beta, Butterflyzer has grown wings -- or for the crossword lovers out there, it's pre-Imago. That's as much as I can say right now, except I'll continue to keep a close eye on its maturation process. I'll also still be leading the Agent Modeling Platform, doing what I can in my off-time to support builds and provide project leadership. The project provides a wide-array of functionality now, and pretty decent quality. Now it is time for people who are interested in moving the ABM effort forward to step up to the plate with their own time and energy. I think the future of tools for modeling real world systems at Eclipse is very bright.

As I said last week it's an exciting time to be building software tools! I know it sounds funny, but I like to think of software development as a helping profession. That's because I think that software products really can help people live more fulfilling, interesting and even happy lives. And I'm delighted to be at Tasktop helping to create those tools.

Butterflyzer Beta

noreply@blogger.com (Miles Parker) — Fri, 06 Jan 2012 21:32:00 +0000

When Butterflyzer was launched a year ago it was introduced with the following words:

"Butterflyzer is in Alpha now. Before we release more polished versions, we need to know if people really want it. So if you like it and want to see a full version, follow us on twitter, and tell your friends. If we get 10 followers we'll release an alpha version. If we get 100 followers we'll release a beta. If we get 250 followers, we'll make a nicely packaged version available commercially at a decent price. Simple as that."

A minor problem.. the response was so great that we achieved the Beta milestone in a few months. And the software just wasn't there in terms of Beta quality. Plus, I felt that there were a number of key improvements to the interface and other functionality that just had to be made. So, it's been six months since we hit the milestone, and I'm happy to announce that..

The Beta is Finally Here!

Download Butterflyzer Now

We've had a Beta Preview available for some time, so many of the features won't be news to all of our users, but I think the level of polish -- if you can talk about a beta release having polish -- is pretty good. There will certainly be issues with it (that's why we have betas!) but it is otherwise feature complete and ready to go to version 1.0. If you're wondering how much work has gone into the release, I'll throw out one measure: the Beta release is Build #995. Here are just some of the many issues big and small that we've addressed along the way..

User Interface

The application just didn't fit in well with the target audience and their expectations about how web related tools should look and feel. One of the most important pieces of feedback I heard was that the interface just felt overwhelming. There is a lot that you can do with Butterflyzer, and I wanted to find a way to make that functionality easily accessible while de-cluttering the user environment as much as possible.

The Butterflyzer Alpha was a typical Rich Client (RCP) application built on the rock-solid and powerful Eclipse platform -- the same one that IBM and other leading software companies base their applications on. The downside of that is that the current Eclipse UI support is a bit stodgy. And unfortunately, while the Eclipse environment provides a lot of functionality, it is very difficult to change the overall look and feel. (E4 makes huge improvements in this area, but E4 didn't address all of our requirements and isn't quite mature enough yet.) So we went back to the drawing board, moved a bunch of stuff around, and re-implemented key existing functionality, including interface and workbench components. It wasn't simple to make these changes, but I think they were more that worthwhile. In fact, I think they were necessary. You can build the most powerful, feature-rich program in the world, but if the UI isn't any good, the only people who use it will be the people who don't have any other choice. So yep, we sweated those single pixels.

Browser Look and Feel

In the quest for simplification, Butterflyzer copies the overall presentation of a typical web browser like Safari, Chrome or Firefox. Catalogs are like browser tabs and all of the interface elements are consolidated within them. That means that we've essentially turned the typical end-user application inside out, with all of the tools and views being hosted inside each editor.

Here's what the old application looked like:

And here's chrome, our simplicity bogey:

Here's the new and improved Butterflyzer:

Why so dark?

By the way, if you've wondered why Butterflyzer employs a "light on dark" visual style, it's not that we're trying to be trendy or anything. There are sound reasons for keeping the Butterflyzer elements dark:

Colors are easier to distinguish, connect and even read against a black background, and color plays a critical role in establishing implicit relationships within and between the various Butterflyzer user elements. Remember those old green and amber monitors? They really were easier on the eye.
A dark background allows images to pop and reduces the sense of visual clutter, which is a critical issue given the complexity of the information visualized.
Most web pages and other artifacts are black on white, of course -- keeping the Butterflyzer elements dark helps establish the web page elements as the visual foreground for the user. There is the "content manager" and then there is the "content" -- everything we can do to make that separation clear is helpful.
Because we insert the semantic tags directly into the web pages, we need a way to distinguish them clearly from the rest of the web content, and we want those tags to look like the tags in the other views. OK, it's not always that attractive, but you can always turn the semantic feature off if you just want to look at the web page content.
A dark background allows us to employ more obvious color highlights for buttons themselves. That's important because most of the buttons are monochrome and a simple as possible; we can then color them as appropriate for user interaction.

Plus it looks cool, right?

New Help System

We've implemented a new Help System that includes dynamic help, so that when you scroll over an interface item, you can get live help. Speaking of help, there is a lot of it! And since we have it now, I'll use it to help tell the story.

Eclipse software developers might be interested to know that the majority of the documentation is actually produced directly from our properties files. (Yes, Butterflyzer is fully designed for Internationalization, but we don't have any translations for other languages yet.) That means that we only have to maintain this information in one place. Custom Java code reads the properties and converts them to Wiki text and the help system dynamically loads the property information for context sensitive help, including links to related documentation. From there, we use Mylyn Wiki Text to generate the Eclipse Help system artifacts, online documentation, and even a PDF manual.

Palette Menus

One major goal was reducing the number of mouse-clicks needed to perform common tasks. I think that number and frequency of mouse-clicks have a large effect on end-user perception of ease of use and efficiency for an application. We needed to reduce the proliferation of toolbar buttons without causing a lot of mouse thrashing. Popup menus work fine for supporting changes in options, but they have a big usability issue. Every time you want to make a change, you have to reopen the menu. You could put options into a tool widget as many Apple applications do -- and we do that for more complex settings involving sliders -- but then users have to get rid of it when they're done. Instead, we created our own menu type that displays as long as the mouse is within the target menu area. This required replacing a lot of the normal Eclipse menu implementation.

If you still think all of those buttons are a bit bewildering, you can find out what they all do here.

Context Popups

If a mouse-click takes time and mental effort, a right-click takes even more. There is just something that feels slow and uncomfortable about the whole interaction. I think that's a major issue with the Eclipse user experience actually, and one of the reasons that people like Web interfaces so much -- you see something, you click on it, it does something. Simple. At the same time, it's even worse to have to mouse up to a toolbar to perform a common task on an item. And the worst thing of all is a hover. There is nothing like waiting for a machine that can do billions of instructions per second to take half a second to decide that you want to do something!

In Butterflyzer, we wanted to support a sense of flow when working with items. And we need commonality between the different kinds of views so that for example the options that you have when working with an item in the graph view are the same as when you mouse over the item in the browser. The solution was a context menu that appears automatically . It's still a bit experimental -- there are some usability and consistency aspects to fine-tune -- but I think it works pretty well. Currently you hold down that "CTRL" key in order to make the menu appear on any item -- having it appear in all cases made interaction a bit busy feeling.

Windows

Yes, Butterflyzer has always done Windows. Just not very well, frankly. Apologies to the legions (well, dozens anyway!) of Windows downloaders out there..we think this latest version will make you much happier.

Visualization

There have been significant strides in this critical area as well, including innovative approaches to taming, filtering, and cleaning up visualizations and related data. For more on all of that, check out this section in our comprehensive documentation.

Internals

The list of enhancements here is too long to get into, but we've put a lot of effort into creating a system that has the highest throughput and integration without compromising data integrity. It's been a challenge, and there have to be some gotchas lurking out there, but I'm really happy with how the system is handling the large volume of data that we can pull in. To see what I mean by "large volume of data", check out this page for details on what you get from a single Butterflyzer search.

Web Site

The web site has come a long way from the alpha days, where it was deliberately minimal and home grown. Please visit!

New Videos

If you want to see how this all fits together, check out the new videos:

Let's Talk!

As always, I'd love to hear from you. What are you using Butterflyzer for? What would you like to do with Butterflyzer?

One thing I'd like to mention specifically for people already involved in developing software tools for the Semantic Web: Butterflyzer is an end-user product, but there are many other opportunities for employing core Butterflyzer technology. Perhap you have proprietary or specialized Web information sources that you'd like to integrate with the Butterflyzer Content Management and Visualization services. Perhaps you'd like to license or re-brand Butterflyzer or make use of key components. Contact us and let's discus how you can use Butterflyzer to enhance your efforts.

But what I really hope is that you're someone who just wants a good tool that you can use for web research, and that Butterflyzer scratches that itch for you. I'm reading the new biography of Steve Jobs so I can't help indulging in a little reality distortion of my own -- in the best sense of that phrase, naturally. You'd never know it with all of the talk, but we're still in very early days of semantic technologies, and we're nowhere near exploring the possibility surface, let alone exploiting it. When it comes to web exploration tools, what exists now for the average user is... well, it's a lot of dumb shit, with a nod to Steve. I believe that you need a rich, powerful application to make the most of the deep store of knowledge that is out there, but it needs to be as simple as it possibly can be -- in short, a tool that opens up the Semantic Web for the Rest of Us. And that's what Butterflyzer is all about.

It's an exciting time to be building software tools! There are a lot of interesting things happening, and plenty of opportunities for all of us to make positive changes in how people around the world work and live. I've got some other really exciting news to share, but that will wait until next week...

Forecast: Spring Clouds and SOPA.

noreply@blogger.com (Miles Parker) — Fri, 23 Dec 2011 02:54:00 +0000

I've been thinking a lot about how to marry the visual appeal and density of representation you get with Word Clouds such as the Wordle and Eclipse Zest Cloudio with the relational information conveyed by graphs. I discussed this a bit in a Recent Post showing Clouds created using Butterflyzer depicting thousands of Tweets from the Arab Spring.

I thought that effort was pretty cool, but the limitations were really obvious. Not coincidentally -- clouds are in the air after all -- Jacob Harris, the guy responsible for many of the New York Time's brilliant infographics (hate the word, but it is descriptive) posted a scathing but quite legitimate critique of word clouds, ending with the line "Every time I see a word cloud presented as insight, I die a little inside." I like this guy's style! Though I think Jacob doth over-harsh a bit. Word clouds have their place. My take is that they are great for depicting the madness of crowds but lousy at deciphering it. As I said in my original post:

"One of the downsides of this approach is that unlike in our graph visualizations, you lose concept proximity. So for example, looking at the cloud you can instantly see that Tunisia and Lebanon are frequently mentioned, but what you can't see is how frequently they are used together. That's what our graph visualization tools do now, and I think there are some neat possibilities for marrying the two approaches."

After I posted this, a couple of my Eclipse compatriots -- Ed MacKerrow and Fabian Steeg -- shared that they'd been thinking along similar lines. I've been playing with this over the last few weeks (probably too much, given that no one has given any indication that it's a feature they're willing to pay for) and I've come up with an initial implementation in Butterflyzer. As always, there are trade-offs and no visualization is perfect, but I'm pretty happy with the results.

A SOPA Opera

Let's use the latest attempt (editorial alert) by the greedy and small-minded to destroy the Internet as our subject. A search on Tweets containing "SOPA" returns a sample of 10,000 or so tweets from the last week. This is what a Butterflyzer produced "Spring Cloud" looks like:

You might notice something right away. See the group of Spanish words to the bottom left? They probably don't have anything to do with the kind of SOPA we're interested in. (Though reading the related Tweets made me hungry.) In a Word Cloud we'd never be able to separate them out of the rest of the clutter, but using the Spring Clouds and Butterflyzer we can eliminate those irrelevant nodes -- along with all of their tweets -- with a few mouse clicks. Hey, we could call that "Cluster Analysis for the Rest of Us"™. (Lame Apple reference numero uno.) Now we can see just the stuff we care about:

Right away we can see how things are associated with each other. Sometimes the associations are obvious; for example when we see "vote" and "house" (bottom middle, in white) together. That's useful, and sometimes you can even pick the content up just by looking at the tags, as in "unconstitutional", "stanford" and "review". But many of the relationships are much more interesting. For example, notice that NDAA comes up quite a bit. When it does come up, it seems to be associated with "@barrackobama, "obama" and also interestingly "#OWS". That gives real insight into the meaning and relevance of the tweet terms, not simply their existence.

We can infer more subtle information than that from the diagram; for example, that the relationship between Obama, NDAA and SOPA in the twitter discourse is organic -- that is, probably not reflecting an orchestrated Tweet campaign, a few viral Tweets or say, Fox News. (If you're looking for those guys, you'll like the video below.) Instead, the use of multiple terms for Obama might indicate that people are associating the president and these two legislative (editorial alert) attacks on the Bill of Rights without any help from say, the mainstream media. And in fact, if we look at the individual tweets, that's exactly what we do see. In the following list we've eliminated the RTs, but the RT ratio to original content is also quite light. If we Focus on those Terms, this is what we see:

Now, if you're someone from the White House or DNC evaluating the potential impact of SOPA on a particular community, it could be useful to be able to get to that kind of information quickly. That's the kind of thing that Butterflyzer was built to do.

The Details

So, how are we creating these Spring Clouds? No secrets or bogus patent claims here. (Perhaps you or someone you know has discovered this solution already; if so let me know and I'd love to add a reference to your work here.) I'll outline the basic approach below...it's really pretty simple. If you're a visualization geek, try to figure about how this was done before reading further. (The name is already a pretty big hint.) If you're not a visualization geek, then good news -- you don't have to figure out any of this. Butterflyzer handles all of the details, and you can set it all up with a few mouse clicks to customize the graph view. See the section below for how.

Preparing the data:

All of the Tweets are collected using Butterflyzer's automated search tools.
Butterflyzer identifies all Terms (words) that appear in any Tweet and indexes the Tweets against those.
We then drop: some common english terms; terms with occurances below some threshold (say 250); and the original search words and any other really common connections.

The visualization algorithm:

Create nodes representing a) Terms and b) Tweet Groups. Tweets are grouped together whenever they have exactly the same set of edges. Edges connect the terms to any Tweets containing those terms. Butterflyzer does all of this automatically based on your specifications.
Layout the Graph using a Spring Layout Algorithm. We use the Eclipse Zest project's Spring Layout Algorithm written by Ian Bull and Casey Best, heavily hacked to provide thing like continuous updating.
The Trick: We then hide the Tweet Groups and their connecting edges.

So, below we can see the graph as it "looks" to the layout algorithm.

And again, here's the same thing with just the Terms:

Now, let's remove the Terms and edges instead of the Tweets:

Each circle represents a group of tweets, and the area of the circle is proportional to the number of Tweets in the group, just as you'd expect. As you can see, we're using a lot of data and a lot of computation for what looks like a pretty simple word cloud.

Curious about what those Tweets actually say? Again, that's easy to find out. Just mouse over one of the nodes (not here, silly -- you'll have to download Download Butterflyzer first, see below) and you can read them, in all their Twittish glory.

So again, this thing might actually be useful, right? Suppose you're a Legislative Assistant for one of the senators who will be deciding whether to vote this (editorial alert) heinous bill out of committee. You might want to dig into what folks that are interested in a particular aspect of the legislation -- such as copyright -- are actually talking about (and perhaps whether your boss or any of his or her campaign donors come up in any of them). You can quickly scan through the cloud and associated terms, and even identify interest clusters.

Limitations

As I said above, all visualizations have limitations. They're on a tradeoff landscape like everything else. Here, the tradeoff vs. Clouds is information density. Because a Cloud diagram doesn't care where words go it can pack things wherever they fit best. We can improve the Spring Cloud layout from where it is now, but there is is a natural limit. Also, the Spring Layout algorithm we're using isn't very good at handling cases of overlapping nodes. Sometimes that's cool, but sometimes it's just messy. Especially since SWT doesn't support alpha for text (but does a great job for shapes as you can see above) the text can be difficult to read and look crowded.

Another issue is that the underlying spring algorithm is in O(n^2). That's geek speak for scary and ugly. As the number of nodes increases -- and as you can see from the examples above, it definitely will increase -- the cost of building a graph increases with the square of the increase in size. There are techniques to mitigate that, but computational costs are a limiting factor, especially for a tool like Butterflyzer that needs to do things like respond to user input in a timely way. We've spent a lot of time trying to tune the UI to be responsive while still putting enough cycles toward the layout, but there is definite room for improvement there.

Finally, compared to Word Clouds, Spring Clouds just aren't that sexy looking. Because we can't just put words wherever we want, things are unbalanced and lack symmetry. We can improve that, but again, only within limits. If truth and beauty have a love-hate relationship, sometimes you're just going to have to pick sides.

Try It!

Want to try it for yourself? Here's what you need to do:

Download and install the Butterflyzer Beta.
To do it yourself:

Do a search on SOPA (or whatever you like) by typing it into the search field.
Click the "Tweet Terms" node under "Collections".
You might want to do some clean up as described above and set up your views to show only the graphs and outline. See the extensive Butterflyzer docs for more on that.
From the Filter Menu, select "Cliques".
From the "Draw Options" menu, Unselect (by clicking) "Show Items", "Show Items Text", "Show Icon", "Scale Count" and "Scale Topsy". Select "Scale Related".

Or just download SOPA Example and open it from within Butterflyzer.

Have fun, and I look forward to seeing your own creations!

Sex, Lies and Spring Cloud Video

Oh, and one more thing... (Lame Apple reference number two, if you're counting.) A nice aspect of the Spring Layout in general and the Spring Cloud is that it is a continuous algorithm, which means that the visualization improves over time, but that also you can add and remove information from the graph without forcing a new layout. This means that we can actually animate the Spring Cloud over time, as I've done in the movie below. Here we're looking at the evolution of Tweets about the Republican Primaries over a span of a week.

There aren't docs on how to do this part yet, but if you want to experiment with this new feature, just select "Select Time Span" from the "Filter" Toolbar Menu or the "View" Menu.

Crossing the ...er... Chasm

noreply@blogger.com (Miles Parker) — Thu, 24 Nov 2011 03:27:00 +0000

Yesterday I went off on a rant about Crossing the Chasm. Within the technology world, that's sort of like mentioning that you don't care for the Bible all that much at a Rick Perry fundraiser. (The book blurb does call it "The bible for entrepreneurial marketing".) Oh yeah, and there's the fact that everyone I've ever asked for advice about making Butterflyzer fly has recommended the book or echoed everything in it. And I'm talking about people that know, that have, like, actually built a company based on its principles.

And so let's be clear right up front that just about every suggestion in the book makes complete and total sense -- not to mention that I must say that this is the only business book I've read -- apart from The New Business Road Test -- that seems to have been written for people who actually like to read and to think. After reading the book I at least now know all of the things that I'm doing wrong. For example, I don't know anything about a) marketing or b) management, but I do know about c) software development. Somehow I figure if I keep doing c), a) and b) will take care of themselves. That's wrong, apparently. I could respond to such reasonable but difficult to implement advice in one of two ways: I could take the advice, or attack the messenger. Guess which option I chose?

Now, the problem with ranting on Twitter is that someone might actually read those rants. Including the rant-ee. Who, in response to my sharing with the world that his book "bugged the shit out of me" and was "one of the more depressing books I've read", very graciously suggested:

Er.. So here they are, my rants, and Geoffrey Moore's all-too reasoned replies. Following that some additional observations, certified rant-free.

Geoffrey Moore: "This article is at the heart of Escape Velocity--Free Your Compamny's Future from the Pull of the Past. That said, I think Steve Jobs is not a replicable phenomenon, so we have to get this done with a "mere mortals" team."

Geoffrey Moore: "I agree. What I suspect we disagree on is what constitutes value-creating innovation because over time it does become harder and harder for established enterprises to succeed with disruptive innovations. In that sense, serial entrepreneurs--the great disrupters--do need to move aside, ideally to go off and do it again via a next-generation start-up."

Geoffrey Moore: "-- Twitter is so the wrong medium for us to have this really interesting discussion--the linearity of any one chasm-crossing does not preclude others, but to be fair to your point, it gets harder and harder for bigger and bigger companies to cross chasms. Bezos is indeed like Jobs in his willingness to make "chasmatic" bets at scale, so when we play choose-up for management teams, you get to start with two all stars!"

Geoffrey Moore: "My final "ouch!" I am torn between suggesting you read Escape Velocity, or just writing me off entirely."

First off, I haven't read Escape Velocity so my responses will be the poorer for that.

I think my critique comes down to two real objections. First, there is linearity and narrowing of markets. The book seems to argue for a very hyper-focussed approach, and given the tiny resources a new company has, that makes a tremendous amount of sense. It's close to indisputable. On the other hand, if a technology is really disruptive, it's going to be difficult to tell a priori just what it disrupts best. Especially if that disruption involves network effects. That's sort of the tongue-in-cheek argument I was making in this post.

I tend to analyze things from the perspective of complex adaptive systems. The problem that jumps out at me when looking at the above approach is that in the ecology of capital, those ideas that are least general and more simplistic will be favored, simply because -- all other things being equal -- those will be the easiest to identify an initial market for. As someone struggling somewhat precariously at the pioneer stage, this is a real problem for me. If everyone demands that one have a market clearly identified, how do I get the resources to identify that market, especially given that the market identification process involves creating enough of the technology to actually see the disruptive effect?

Along with this there we have the observer effect. We have certainly seen that the companies that succeed in the venture capital arena closely follow the strategies spelled out in Crossing the Chasm. But do the companies succeed because they have the best strategy ex-ante or because they are most co-adapted with the culture of venture capital, which is heavily invested in the Chasm model? (And I fundamentally don't buy the naive classical view that some capital sources will evolve to fill the open niche.)

To whatever extent the above arguments hold, we're missing out on a lot of possibilities. To that extent, any really transformative technological innovation would then come from large companies willing to take risks on complex and general technologies. And when you look at it, that's exactly what Apple and Amazon are doing. They're hiring and finding ways to value the actual innovators that might have been working in a garage somewhere twenty years ago. (Do we even have any good examples of the garage model in the last few years? I can't think of any, but that doesn't mean they don't exist.)

Now, to the extent those arguments don't hold, you'd want to do everything that the rest of the book suggests. And I'd stipulate that in the very large majority of cases, that would in fact be the right thing to do. But we shouldn't forget that innovation is the canonical fat-tailed phenomenon. What that implies is that the potential benefit from the minority of cases where real innovation doesn't fit into the Chasm model box could very greatly outweigh the benefit from those that do.

But the fact is that -- back in the real world -- we should assume that they even if they don't hold the observer argument means that any potential innovative company must behave as if they do. And yes, that's the part that bugs me, because it does seem to have depressing consequences. Because it means that you don't get to have a company that is truly driven by a core belief in innovative product and innovative people unless you already are Jobs or Bezos. And you don't get to work in one unless it's Apple or Amazon. (I realize I'm making an extreme case here.)

I think that explains why those companies are able to attract people who actually have something unique to contribute. The obvious question to ask is "why aren't there more companies like that"? And especially, "why aren't there any [ok, very many] non-gigantoid companies like that"? No, I don't think it is simply because of the unique, raw talent of these two CEOs and that no-one else has the vision, passion or creativity to do the same thing. I think it's actually because no-one else has had the balls and the money to do it. Don't forget, Steve had to spend a decade in exile because venture capital thought that selling sugar-water was the best model for a 'mature' technology company. The real question is "what is it about the current setup that prevents other companies from doing it?" "What did Jobs and Bezos overcome that others haven't been able to?" In my book, a large part of the answer is that what is in the way is actually "a culture that commodifies ideas, driven by people looking for the simplest, most repeatable and predictable business model possible".

Here my concern is actually not so much with the serial entrepreneurs -- as suggested, they can more or less fend for themselves. Now, some of those people will also be the true innovators, but my sense is that (especially in the web space) more and more of those people are actually closer to professional managers and tech-savvy investors than passionate innovators. My concern is for the people actually doing the innovation. And it turns out that, perversely, they are on the losing end of the stick any way you look at it. And so this was the part that I found so depressing. Not because it isn't correct, but because it probably is:

"In the development organization, pioneers are the ones who push the edge of the technology application envelope... Their brilliance fuels the early market, and without them, there would be no such thing as high tech.
Nonetheless, once you have crossed the chasm, these people can become a potential liability.. In the meantime, they are not likely to cooperate in the compromises needed, and can be highly disruptive to groups seeking to carry this agenda out. It is critical, therefore, that as the enterprise shifts ... that pioneer technologists be transferred elsewhere..if necessary to another company." [I think the word we're struggling for here is "fired".]

"The final world on pioneer technologists, I suppose, is that they are in the same bind as authors -- a fate I can identify with. Like authors, they are compelled to conduct their craft regardless of whether anyone will pay for it. As such their negotiating position is fundamentally weak, and their normal compensation reflects it."

Let's be clear: these are the very people without whom none of the innovation that drives any company growth can or would have occurred in the first place. Geoffrey acknowledges that "casting aside people, dislocating their lives and threatening their livelihood, is immoral", but I can't help but read that as an awkward admission of an inevitable consequence. And I wonder how many technologists know that this is very literally the playbook that their company management and funders are using, and what they would do if they did? I know that I would neither want to join a company or to hire people for a company that thought this way. In fact, my experience is that many companies -- certainly the ones that I admire -- don't really think this way: most people simply aren't that venal. Rather, it seems more of a kind of macho posturing for the benefit of the funding community. Still, isn't that kind of sad?

So at core, I am not making an economic or business argument, but an ethical, even spiritual, one. I'd like to see people living happier, more fulfilled lives. I guess I should be embarrassed to admit that, but there it is. Unfortunately, we live in a world where business concerns are orthogonal and often actually opposed to human concerns.

Happiness requires a sense of community, of shared responsibility, common interest and diversity of ideas and styles. Healthy dynamic systems need that kind of diversity, that soft-permeability, and they need it in day to day interactions, not just through the periodic purging process demanded by investors as a result of moving to another stage in the Chasm life-cycle. If you look at any social system -- with the notable exception of those constructed via the crushing, simple-minded logic of modern capitalism -- that's exactly what you see. Why can't we build companies of all sizes where people from all ends of this spectrum co-exist? Wouldn't that be a worthy goal, or do we as always fall back on the argument that business is about what is, not what we would like it to be?

The Arab Spring, Zest Cloudio and Butterflyzer

noreply@blogger.com (Miles Parker) — Wed, 05 Oct 2011 00:33:00 +0000

One of the goals that I'm really trying to drive Butterflyzer toward is finding a middle ground between the level of detail that hard-core social network analysts get into, and the incredibly over-simplified executive dashboard approach that so many of the social media analysis startups are providing. A big part of this is providing fluid, interactive visualization. So far a lot of that has been based on graph visualization, but we're also experimenting with integrating other approaches.

One of the cool things that we've got lying around to play with are a bunch of tweets collected during the start of the Arab Spring. We've used a lot of different techniques for visualizing and analyzing this data, but when the Eclipse Zest Cloudio toolset came along recently, I thought it would be an interesting way to quickly see what exactly people were really talking about. I think the results speak for themselves. Thanks to Stephan Schwiebert for this great contribution. One of the downsides of this approach is that unlike in our graph visualizations, you lose concept proximity. So for example, looking at the cloud you can instantly see that Tunisia and Lebanon are frequently mentioned, but what you can't see is how frequently they are used together. That's what our graph visualization tools do now, and I think there are some neat possibilities for marrying the two approaches.

Butterflyzer work has kicked back into high gear, and I'm currently collecting live streaming data from Twitter on the OccupyWallStreet movement. I can't wait to put together a similar word cloud for it. Obviously, a lot has changed in Egypt in the nine months since this data was collected. Who knows what will have changed here nine months from now?

Methodology: The data was collected from 10,000 tweets between Jan 23 and 29th containing the word "egypt". We then extracted word counts, removing common english words and other noise, and then visualized the remaining words with Cloudio, sizing for frequency of use.

Eclipse BFDT Project Proposal

noreply@blogger.com (Miles Parker) — Fri, 09 Sep 2011 22:02:00 +0000

Hi all. I wanted to let folks know about a new project proposal I'm considering, and ask for your feedback and support. I'm really hoping you'll consider participating signing on as an interested party or mentoring this effort.

As you can see from the proposal, I've removed certain identifying features from the initial proposal. As we've seen in other recent cases on the Open Source community, project branding and naming can sometimes present certain difficulties. If you promise that you are not easily offended, you can find out more about the project by visiting the project proposal page. What do you think, worth submitting?

I put the proposal together in an hour or two, so to everyone who says that the Eclipse process is too heavyweight -- just shut the ____ up, ok?

Update:Someone already beat me to it. ___.

Mylyn + Your GMF Models = Love?

noreply@blogger.com (Miles Parker) — Thu, 08 Sep 2011 20:00:00 +0000

As a quick follow-up to my post yesterday, here's an overview on how to integrate Mylyn with your own GMF editors. We'll be maintaining this on the Mylyn for Modeling wiki page but I wanted to share it with you here as well so you can see how easy it is to get the Mylyn love into your diagrams.

There are a number of artifacts to take care of but much of it is boiler-plate. (It would be straightforward to support this with a simple meta-model and code generation, hint, hint..) Please let us know if you have any difficulties or improvements for the below steps.

The Basic Idea

Users aren't interested in the diagram objects, they're really interested in the underlying domain or model objects. In the Mylyn Java integration, for example, Java classes and methods are tracked, and then revealed in various ways in different editors. So for an EcoreTools model for example, we don't want or need to keep track of the diagram Node objects or any other reference in the various foo.gmf.. files for, we want the domain object itself, e.g. EClass, EPackage, etc.. in the corresponding foo.ecore resource. What we need to do is to define what the user might be interested (or not interested) in and then map that to the diagram editors and other editor types.

Your Turn!

The best thing to do is just start with one of the two existing projects. We'd recommend org.eclipse.mylyn.modeling.papyrus.ui. org.eclipse.mylyn.modeling.ecoretools.ui will give you some ideas about how to work with a more complex scenario.

Refactor Artifacts

Rename files and other artifacts with the appropriate names. For example in your GMF "Foo" Diagram tool:

Uml2DiagramDecoratorProvider -> FooDiagramDecoratorProvider

In Plugin.xml

<decoratorProvider class="org.eclipse.mylyn.internal.modeling.papyrus.Uml2DiagramDecoratorProvider">

Becomes:

<decoratorProvider class="org.you.foo.FooDiagramDecoratorProvider">

etc..

Here's where the non-boiler plate stuff is.

Structure Bridge

In FooStructureBridge, you'll define domain objects which should be managed by Mylyn. This is how it is implemented in UML2StructureBridge:

@Override
 public Class<?> getDomainBaseNodeClass() {
  return Element.class;
 }

 @Override
 public Class<?>[] getDomainNodeClasses() {
  return new Class[] { Classifier.class };
 }

 @Override
 public Class<?> getDomainBaseEdgeClass() {
  return Relation.class;
 }

 @Override
 public Class<?>[] getDomainEdgeClasses() {
  return new Class[] { Relation.class };
 }

See DomainModelContextStructureBridge for API details .

Diagram Bridge

In FooUiBridge, you'll define diagrams and views which should be managed by Mylyn. In UML2UiBridge, the important bits are:

@Override
 public boolean acceptsPart(IWorkbenchPart part) {
  return part instanceof PapyrusMultiDiagramEditor;
 }

 @Override
 public boolean acceptsViewObject(Object domainObject, Object part) {
  if (domainObject instanceof Classifier) {
   return part instanceof ClassEditPart;
  }
  if (domainObject instanceof Package) {
   return part instanceof PackageEditPart;
  }
  //Edges
  if (domainObject instanceof Relationship) {
   return part instanceof ConnectionNodeEditPart;
  }
  return false;
 }

See DiagramUiBridge for details.

Simple, Eh? Just a lot of search and replace. If you want to contribute or help with code generation for this, that would be a really nice...

So try it out and please let us know how it goes or if you run into trouble. It would be awesome to see more complex examples of Mylyn based GMF model editors, so please experiment and share your examples! And don't forget to check the Wiki Page as that's where the up to date documentation will be.

Mylyn + Modeling = Love

noreply@blogger.com (Miles Parker) — Wed, 07 Sep 2011 00:46:00 +0000

It's been a while since I've posted anything here. Been busy doing real work -- you know, the kind that other people pay you to do. Specifically, I'm delighted to be working with the team at Tasktop, where I've been focussing on taking all of the Cool Stuff™ that Mylyn does and making it work with Eclipse modeling tools.

This looks like a match made in heaven. Eclipse is a great IDE, but if one were just looking for an IDE, there are many alternatives. But I think I can safely say that the Eclipse ecosytem has no real competition. What makes Eclipse truly great is not the bundled Java development tools, but the powerful and highly adaptable platform design, and most importantly of all, the open, transparent, and quality-focussed community structure that the Eclipse Foundation has worked so hard to build. That allows anyone with a good idea -- and the resources to make it a reality -- to play in the same sandbox as everyone else. For me, there are two exemplars for the potential of the ecosystem to be much more than the sum of its parts.

The Eclipse Modeling Tools, including EMF and related tools, are what brought me to Eclipse platform in the first place. If you want to do real-world Model Driven Software Development, it would be hard to make a case for anything else.
Mylyn demonstrates the kind of practical invention that is possible when a platform is really open. It provides a unique task-focussed approach to development. Admission time: It takes a bit of up-front investment to really grok Mylyn, and I didn't take the time needed to really integrate it with my day-to-day development effort until I started working with Tasktop. Working on my own and setting my own tasks, I just never really thought I needed it. But now that I'm back working with a team of developers on a regular basis, with dozens of tasks to keep track of, I'd be seriously annoyed not to have it.

In many ways, the match was remarkably -- sometimes ridiculously -- easy. It is a really cool experience when you work with completely different technologies that have been designed for adaptation to arbitrary uses, and those designs actually work! In some cases, such as making the Ecore Tools project manager integration work with the modeling context bridge, the actual implementation (minus all of the project packaging, .xml bits, etc..) was only a few lines of code.

Still, the connection didn't seem quite so natural when the two got together for a first date. David Green had put together something similar a couple of years ago, so I knew it was possible. But would sparks fly? Text and tree editors are naturally places for approaches that constrain and contextualize data selections. Graph structures are even better. The problem is that most diagramming artifacts aren't really graph structures -- they're actually pretty (or let's be honest, usually not so pretty..) pictures of graph structures.

The typical approach to software model diagrams is to take a snapshot of entities and their relationships and stick them on the wall. And people get really attached to where different elements are on that wall. In fact, I think one of the greatest anti-patterns for graphical modeling tools is to treat the pictures as the most important artifact of all...but I'm getting off-topic. The important point is that for the common types of diagrams such as UML class diagrams and Ecore we can't really move things around. Or we can, but then they wouldn't be the same artifact anymore. Again, that's because with many diagram tools the layout and location of various entities have semantic meaning, or at least people treat them like they do. And in practice, GMF doesn't provide a facility for changing any aspect of the actual diagram without affecting the persisted model. That seems like a no-no for a tool like Mylyn which is intended to be assisting developers, not modifying artifacts.

So clearly a good dose of gentle persuasion and hand-holding was called for. And I discovered that what GMF does have is support for decorators. Decorators essentially allow you to annotate a model without affecting the artifact itself. Rather than mucking about with the layout, could something workable be accomplished with creative decorations? The idea that made the most sense to me was to mask out the parts that aren't interesting, unmask or highlight pieces that are, and then sort of "ghost in" the other bits as people navigate the structure. I think it worked out pretty well, but I'm anxious to hear what you think. Does this approach make a decent trade-off between hiding the information you don't want to see and making it easy to discover the information that you do?

Those who have worked with the GMF Runtime may not be surprised to hear me say that the mechanism for doing all of this didn't seem exactly straightforward. You know how some frameworks seem never to have met an abstraction that they didn't like? There are probably great reasons for every one of the levels of generalization provided by the decoration and related mechanisms, but taken together there were a lot of pieces to grok. Thankfully, the EMF Compare team had just done a similar integration so I was able to learn a lot from that. But with a few false starts and uncomfortable moments, I can happily report that the couple is doing quite well, and I'm happy to have played a small role in this little bit of bliss.

It's not quite "official" yet, but there should be a contribution to Eclipse quite soon. In the meantime, it's reasonably stable and works with Ecore Diagrams and Papyrus UML Class diagrams. Details are here and we'd appreciate your bug reports and especially ideas for improvements.

Oh, ok... while I was at it, I grabbed a video of the two in action. Was that wrong of me?

The Indolent Development Manifesto

noreply@blogger.com (Miles Parker) — Thu, 23 Jun 2011 19:42:00 +0000

Every once an a while, a new way of thinking comes along that is so powerful, so unique, so practical -- just so damn cool, really -- that it catches on naturally with no real effort on the part of the innovators who created it. A decade or so ago it was "Extreme Programming" and then "Agile Development". Remember? It seemed like a crazy idea one day, then something you had to spend months talking your boss into the next day, and then the next it had become accepted practice. In other words, totally not-cool. You ended up feeling like that annoying guy who says that REM only went downhill after Murmur, and that he can't even stand to listen to any of the later stuff. Then you realize that that anecdote only makes sense to 20% of your readers. Then you feel like that old guy smoking by the fire exit, talking about how he could have done any of that so-called Business Process Management just as easily in COBOL and it would have been more maintainable. Where was I going with this?

Oh yeah... today I wanted to introduce to you a new paradigm for Software Development, one that seems as radical now as Agile did back in the day, but that fits in so well with the way that we really want to do software development that you'll wonder why it took someone so long to write it down. (I'm wondering the same thing myself right now.) In fact, you may already be engaging in some of the practices without ever having really bothered to think about it.

Before introducing Indolent, I need to give credit where credit is due. I'd be more than remiss if I didn't acknowledge the work of important innovators in this transformative approach. I'm speaking of course of Ken Laidback, and his LaZy-Ass Programming approach. For those who haven't been keeping up with the hot trends in software methodologies for the last couple of years, well, you've got the right idea. Just keep being who you are. But most of us need a point in the right direction -- a formula of some kind -- and that's what Ken and his merry band of hacksters has given us. In a nutshell, LaZy-Ass Programming -- later referred to without that cute capitalization in a failed bid to satisfy those uptight "Suits", the people who are always looking around for shallow opportunities to make "money" -- was an attempt to solidify and round out a set of behaviors that the initial practitioners had found made them happiest. Rather than a set of "Processes", or even "Practices", Lazy-Ass was at heart a set of "Excuses" and "Rationalizations" for behavior that fit in with the way programmers really wanted to do things anyway. But the real beauty of Lazy-Ass was not the Rationalizations themselves, but they way they all fit together into an absolutely water-tight container capable of keeping the practitioners warm, dry and responsibility free, happily doing "Cool Stuff" -- defined as "what you want to do" as opposed to "what the Suits think you should do" -- for years to come.

A couple of examples should give you a flavor for how these interlocking rationalizations work in real life. "Task Shuffle" invites practitioners to take on unrelated random tasks and then swap them with others at random intervals so that the Suits are never really able to identify who is responsible for what. "Goat" gets (almost) everyone together in an effort to identify one or two people on the team who will take on responsibility for those few unavoidable cases where "Task Shuffle" fails -- by concentrating blame, the rest of the team is able to focus on "Whatever the Hell I Want", thus doing Cool Stuff. "Welcome to the Team, Let Me Buy You Lunch" ensures a steady supply of new recruits to take on the Goat role.

Most of us are familiar with the rest of the history. Lazy-Ass managed to establish itself in a number of key domains, most notably government contracting and systems integration. Here, the interests of the programmers and the rest of the stakeholders (excluding the "Customer", whose role is largely secondary, and some would argue purely mythical) strongly coincided. But it never caught on in the mainstream. I'm going to say something a bit controversial here. I think the real reason for this was mostly packaging. Yes folks, how you present things matters! Lazy-Ass Programming got a bad rap because of its name and the over-enthusiasm of early practitioners. "Lazy-Ass" was simply to* cute and -- let's face it -- provided a little too much of a clue about the deeper methodological agenda for those executives concerned with external appearances and shallow concerns like "income". Later innovations like "Rats and Ships" and "Bigger-Better-Deal" convinced some junior execs to buy in, but it was a matter of too little, too late. And yes...let's admit that some of the early practitioners were just a little too dogmatic. Let's be realistic, you've got to throw the dog a bone once in a while. Still, even if not in name, the spirit of Lazy-Ass lives on in all sorts of efforts and at all sorts of organizations, big and small.

Some of you may be aware that as of this writing, the Eclipse Project Indigo Release just shipped. This project involved the simultaneous release of 62 separate projects and...um, a bunch of lines of code but I'm too lazy to look it up, and LOC is a lousy metric anyway...all at once. This sets a terrible precedent for those of us who simply want to do "Cool Stuff" and who are above caring whether people can or will actually use it. In fact, I can't think of a more colossal failure for the Lazy-Ass approach. That's why I've chosen to introduce my new methodology now. If it takes hold, perhaps we can prevent a worse atrocity next year.

My new methodology is called "Indolent Development", and I own the trademark so don't even think of borrowing it without giving me attribution and a nice cut of the action. I'm going to replace Agile -- everyone knows that that's totally broken anyway -- and in the process make a million dollars. Yes, that's right, training courses are in development right now, and soon you'll be able to get personalized coaching from a highly-trained, experienced network of Indolent Masters. The major thrust of Indolent is to be much more "CIO friendly". Even the term "Indolent" was carefully chosen to fly under the corporate radar while amusing those of us "in the know". (You'd be surprised at how obtuse many corporate leaders are.) Indolent Development involves just ten core principals. These principals* are designed to maximize Cool Stuff, allow plenty of Slack, and reduce or eliminate exposure to Downside Risk for the practitioner.

YNKWYMNI "You Never Know When You Might Need It": If it seems kind of cool, put it in! You can always take it back out later.
YBIYFI "You Broke It, You Fix It": This one is all about Shared Bug Ownership. Example dialog: "It worked fine when I wrote it. I've moved on to something much cooler now. YBIYFI!"
The Random Walk If you're really thinking outside of the box, a plan only gets in the way. In fact, thinking ahead at all is a sure way to kill innovation.
Interrupted Integration Example dialog: Hey, this worked last month. WTF?! Look, just give me a couple of weeks to sort this out, and it will probably be working great for the next build. Unless someone breaks something else in the meantime.
Refutzing Let's take everything apart and put it back together. Maybe it will work this time.
Users Lie They don't know what they really want. Why waste time asking them?
Test Later Why bother fixing it if you don't know its* broken? Real programmers have confidence. (It's important to note that testing does serve a purpose. "Test Weenies" make good "Goats", because you can always point out their failed tests.) An advanced form of this practice is Test Never.
Eight, eight, I forget what eight was for.
Self-Expression You're special. Why shouldn't your code reflect that? When you die, won't it feel good knowing that somewhere buried in that nuclear reactor code there is a special little nugget that only you can really appreciate?
No Best Practices When a good idea becomes dogma, anti-dogma becomes a good idea.

And actually, if you take nothing away from this article, that last point is it. In fact, you could say that it is the only point that really matters.

*Hey Mr. Suit, programmers can't spell and frequently confuse homophones. Get over it.

Update: I added a bunch of cool pictures so that my blog is just like Ed Merks'. Except I was too lazy to take my own pictures. So I stole some off the internet when I was supposed to be working on software builds. Also, because I'm working on builds yet again, god-damn-fuck-it-all-to-hell, I'm kind of in a bad mood, so they're all pretty depressing and uninspiring. Sorry. I hope your day is going better, sincerely.

Image Credits:

Guy who is almost as cool as I am, but someday will also be just like COBOL guy, Wiros
Nose Picker, Millicent Bystander
Lazy Sign, Jef Godesky
The Most Depressing Place on Earth, After Fire, Kevin Dooley
Meeting Interessante, Antonio Furno
Just like my EclipseCon talk, Out.Of.Focus (Why do Flickr people not have real names?)

Butterflyzer Alpha 5

noreply@blogger.com (Miles Parker) — Thu, 16 Jun 2011 04:08:00 +0000

When I first thought about turning Butterflyzer into a product, I really had no idea whether people would grok it or not. It seemed to me that if I couldn't at least get that hard-core group of "pioneers" to take a look and find some value in it, it really wasn't worth doing. Now, Butterflyzer has about 75 followers on Twitter, which is getting frighteningly close to 100, the point at which we've said we'd deliver that promised Beta. So what to do now? Why, add a whole bunch of features / potential defects, of course! In this case, these are features that I've been wanting to build in from the beginning, but they've also been driven by the survey feedback we've been getting.

One request had been to provide Butterflyzer as an Eclipse plugin. Done! This means that if you're already using Eclipse you can fully integrate Butterflyzer with the rest of your workbench and workflow. Just head on down to the Eclipse Marketplace and grab yourself a free alpha! (But remember, it is Alpha software and you might end up with that dreaded spinning wheel of death, or worse, so you might not want to keep it in your main production environment just yet.) After installing the plugin, just open the Butterflyzer perspective, create a new general project, create a new Butterflyzer model using the wizard, open it up, and type a search phrase into the toolbar text box. And then see what comes up!

One of the least expected feedback items I've received is that while many people find the graph visualization to be really compelling, a lot of people don't see much obvious value in it -- they often see the real value in the organization and curation capabilities. I do think there is a huge potential for dynamic graph visualization in allowing people to understand deep semantic relationships, but I also am getting the idea that the graphs are not the point -- the information is. Tools like Mylyn / Tasktop have done a great job of putting the graph behind the scene without losing the representational power that the graph structure provides. And that's what this release has been focussed on... making it easy to navigate through the information you want in a very direct way, without losing the rich structure behind it. As this vision gets built out, hopefully the whole idea behind "Think Outside of the Browser" will become more clear.

To facilitate this, there have been a number of improvements to overall usability. Elements having the same connections are grouped together in a nicer way..

And toolbars and captions are better integrated with the graph visualization..

You'll notice that the web pages in the captions are automatically pared down to grab just the important details from the site. OK, that part needs some work, as you can see below, but it already better than trying to scroll through a little page with a bunch of ads and image links. By clicking the toolbar, the caption expands (with a nice animation) so that you can see the full web page...

But the most important feature is that Butterflyzer now becomes a very powerful semantic browser. Notice that embedded in the page are buttons for semantically tagged items. Tags are created for semantics that OpenCalais has recognized, search terms that you've used, and any other categories that you've created. When you click on one of those items, you'll see all the items that relate to that tag within your browsing context. For example, we've clicked on a prominent Eclipse personage here, and even though I can't spell his name, I can easily grab all of the articles mentioning him...

Now of course, if I was using an ordinary browser, I could have done a new search for "Chris Ani.. Eclipse Indigo", but that would be a) unwieldy (I'd copy and past his name from the list), b) slow, and c) I'd have to look at every page in turn to find out whether I was interested in the contents. With the revolutionary (sorry, couldn't resist) Butterflyzer Tile View, all of the matching pages that can fit comfortably on a single tile are loaded simultaneously.

Form the Tile View, I can expand up full pages, click on the semantic tags within them, return to the graph, etc.. It's difficult to get a sense for how fluid this is and for the potential of the approach overall without trying it for yourself.

You can do the same thing with Tweets -- here we've selected a set of key Eclipse sources commenting on the Indigo release...

By the way, we're looking at beefing up the Twitter component -- it's pretty low hanging fruit actually -- so that you can do all of your tweeting directly from Butterflyzer, but we do have to tread a bit lightly here to stay off of the "Basic Twitter Client" category -- Twitter has made clear that that is their territory. So how about it? If you think a really powerful full-featured analysis-oriented Twitter client should be part of the Butterflyzer feature set sooner rather than later, let us know.

Hopefully that's given you a sense for what's new with Butterflyzer. Check it out, install the plugin, and let me know what you think!

Oh yeah.. you can download it from here. The plugin update site is http://butterflyzer.com/downloads/alpha/update or just use Eclipse Marketplace. Please note that the Butterflyzer plugin works with Indigo only (unless we get huge demand for legacy support) -- just one more good reason to download Indigo!

Butterflyzer Alpha 4

noreply@blogger.com (Miles Parker) — Tue, 17 May 2011 22:22:00 +0000

We've just released the latest Alpha for Butterflyzer. If you're not sure what Butterflyzer is, check out this article and this one. You can download the free alpha here. Most of the changes in Alpha 4 have to do UI improvements and overall user experience. First, we've dealt with some important quality issues. Yes, this is Alpha software, but people should be able to use it with little pain as possible!

Fixes

Twitter Authorization: We've finally laid to rest some really pesky issues. For very good reasons, Twitter wants to make sure that we use their authorization screen, which means that we need to deal with the inevitable changes that happen on their end. Still, there is nothing worse that getting a bunch of traffic to your site only to discover that your software doesn't actually work because something in the external environment has changed between the time you built and tested it and when it was deployed. OAuth is not the easiest technology to deal with, but it ensures that users have the last word when it comes to their accounts.
UI Freezes: I also think we've crushed the main issues with synchronization, aka "the dreaded spinning ball of eternity". Butterflyzer is doing some pretty bleeding edge stuff with EMF and Zest visualization updating. An important part of the overall design has been to grab information in parallel so that while one process is waiting for a request to be serviced by a REST or another query somewhere, we're busy collecting all of the information we can from other sources. If you try it out I think you'll be impressed with just how much information you can suck down in a very short time -- we're using a lot of straws! So, we can have dozens of separate processes running at once, and they're all updating a live EMF model editor and Zest view, while the user is also interacting with the model! That's a perfect recipe for concurrency issues -- especially since according to Ed Merks, EMF isn't actually intended to be thread safe -- and while things have gone surprisingly well, we've had our share. When the UI locks up, that has a definite impact on quality perception! The way we've handled this is not with a lot of thread blocking and synchronization calls. In my experience, that's a recipe for endless hacking. Instead, my strategy has been to treat the visualization in a very fail friendly way, while focussing on protecting the model from anything that happens in the UI. The last set of issues were due to the way EEF was handling updating. They couldn't have anticipated our usage and it was very easy to override the default behavior by swapping in a custom base component.
Other stuff: A lot of little glitches and annoyances have been taken care of. If you have some 'favorites' that you'd like looked at, now is the time to start letting us know.
Unresolved: The Google Insights, Exhibit and Timeline stuff is still really brittle. There are a bunch of time-consuming issues to resolve for that to work seamlessly.

Now on to the fun part...

What's New?

Graph Visualization: Many improvements here. Some are subtle, but others dramatically change the feel and functionality, I think for the better. For example, grouped items are now all placed into a big text block. This improves performance and readability. There is still a lot of experimentation to do and as you can see below, we'll be giving you more and more control over how the visualization is done. If you've got feedback on any of this, please send a note.
Performance: Lot's of little things, especially in how large searches are handled add up to a more fluid experience. The "full Butterflyzer" -- Google Search, Twitter Tweet and Author Search and term processing, Topsy statistics and trackbacks and Semantic enrichment of 60-100 separate web pages -- is finished in just a couple of minutes, with the key results available right away. By the time you are interested in the more complex stuff, it's already there. For comparison, when I first started experimental implementation of the tool that could take as long as 45 minutes. And if you tried to do all of this manually, you'd be at it for weeks.
UI Polish: A lot of little improvements. There are still rough edges, but we're making significant progress. As one simple example, things are now put in a sensible order in the tree editor. Moving items among collections works better now too.
Graph Customization: I saved the best for last. Perhaps you thought the Butterflyzer graph colors and design were hard to read or just plain ugly. Well, now you can customize all of that to your hearts content! Just click on the "Butteflyzer:Preferences" menu item, and navigate to the Visualization page:

Here are some examples of what you can do:

So that's what's been happening in Butterflyzer land. If you're using Butterflyzer, I want to ask one important favor. Please fill out the Butterflyzer survey. I know people are using the software, we have the downloads and site visits to prove it! But we've only had a couple of survey responses. Those have already been helpful in identifying key issues. For one thing, it's convinced me that the number one thing I need to do is to help people understand what makes Butterflyzer more than just a fun tool to play around with. I need to show people how it can be really useful in their day to day work, and that's what I'm going to be working on next. In the meantime, I'd love to hear from you about what you're done with Butterflyzer. If you have visualizations or interesting data sets you've collected with it send them to me at miles @ [our domain name].

I'm going to drop another plug in here. We've gained a lot of experience at this whole social media analysis thing, and I think we have some unique capabilities to offer, including consulting and custom development. In addition to our expertise in Social Network analysis, computational modeling, visualization and artificial reasoning, our colleagues have literally written the books on genetic algorithms and other optimization techniques. If they can't figure it out, it's not likely that anyone else can. So if you or someone you know have really challenging problems, and you'd like to get out ahead of what everyone else is doing, please drop me a line at the above address.

And don't forget the survey..!

Why Model Reality? Science, Eclipse, Big Data, AI, and what's missing...

noreply@blogger.com (Miles Parker) — Fri, 06 May 2011 02:03:00 +0000

Last month I gave an extended talk at EclipseCon 2011 about the Eclipse Agent Modeling Platform and the whole idea of Modeling Reality in general. Why would we want to do such a thing? What does it have to do with software development and Eclipse? Well, according to one comment I just received about the talk, not much. My talk was:

"Too scientific. Just for special people"

I thought that was a very fair and amusing reaction. Many people do come to EclipseCon expecting to find out only about information that they can use to do their day-to-day work and are understandably more interested in practice than theory. I can tell you that when I walked into a room large enough for 80 people and there were maybe 20 people spread randomly around the room, I felt more than a bit awkward. I do appreciate the EclipseCon program committee giving me the opportunity to talk to people about what might seem to be esoteric topics. And you know what, my talk actually was pretty relevant, given that we had keynotes on Hadoop (Big Data) and Watson (Big AI) and I pretty much spend half of the slides taking swipes at both.

So I was thinking, aren't there a lot of special people in the world? If you're intrigued by any of these assertions disguised as questions...

What do Egypt, Watson, Teenage Pregnancy, Big Data, Nuclear Reactors and the Eclipse Modeling Framework have to with each other?
Why is Big Data Big Dumb?
Why is AI just Smart Dumb?
What kind of reasoning approach is just as valid as Induction and Deduction?
What do Tasktop, Butterflyzer and other Eclipse RCP apps have to do with this?
What does this have to do with Software Development?
What does this have to do with me?

..then watch the video and find if you're special too!

I've left out the demos; I'm going to put them into a separate video. I've also left out the architecture slides because you can find them elsewhere and they're incredibly boring even to me.

[You'll need to click directly on the blog to see the video.]

The Eclipse Community and Victory over Conflict

noreply@blogger.com (Miles Parker) — Thu, 05 May 2011 21:41:00 +0000

I just read Andrew Bayer's really nice post. I tried to respond there, but naturally I got too verbose. Someone needs to come up with a good solution to this problem of blog posts and comments and half-blog posts half-comments that end up spread all over the place. And make a hundred million dollars on it. (And no, Google Wave wasn't it.)

Andrew said: "It's a natural human instinct to want to be the winner when there's a conflict, but no reunion is going to be possible if one side insists on a PR victory over the other." That's really really true. It is always necessary for people to give up ground. But I think that's what we need to do and I'm going to talk about that in a followup post.

As I mentioned in my earlier post on the subject, the thing I really appreciated about Jenkins is the creative way that they found to step outside of the conflict altogether. I mean maybe that wasn't entirely the intent, but it felt like someone had read The Art of War. Which is just as much about peace in a non-obvious way. I also appreciate what Andrew had to say about sometimes things just going there own way.

But if Andrew and the rest of the Jenkins community are interested in reaching out or engaging in any way, I know that there are many people in the Eclipse community that would be happy, actually eager, to help. I think very few people really want to see this as an us-vs-them thing.

But how to deal with it? I'm going to say something unorthodox. I don't think you really make things better by talking about them and negotiating and making peace offerings. So often that just makes things worse. People's positions just get hardened further, and we create more history to overcome and more confusion. We see that in the Middle East and Korea and that is with all kinds of experts helping to "fix" things. And you certainly see that in this case, where people are still arguing about posts that someone made six months ago.

So from what I've seen some kind of meeting in the DMZ and coming up with a reconciliation plan often doesn't work out that well. Instead, I think it works better when people get together in an organic way or just get fed up with the hassle of arguing all of the time. This isn't a top-down leadership process, it's totally bottom-up, people talking to people. In this approach, disputes aren't overcome, they simply become irrelevant. One day you just wake up and think "what the hell was all that about anyway?"

So to get there doesn't really require Oracle or Jenkins to make some kind of specific gesture. And actually, come to think of it, what's Oracle? Oracle doesn't really exist. (And neither does Jenkins for that matter.) It's just a bunch of people, contracts, equipment and so on that it's convenient to pretend is a single entity. So let me tell you that the great thing about Eclipse is that it has a sneaky, even subversive, way of separating out people from corporations. (Don't tell the suits.) It is about building trust in and relationships with other developers. So practically speaking here are some paths that Jenkins could follow. They aren't mutually exclusive!

1. Continue to build the great community and technology that Jenkins now has.

2. Indicate that you are an "interested party" for the Hudson proposal. Perhaps request that Kohsuke be made a project co-lead. All communications about projects at this stage are supposed to be public. But note that part of the Eclipse Proposal process is a loose mandate to projects to make an effort to bring in lot's of different efforts. You can't force that process but if everyone is acting in good faith it could work out nicely. Of course there are lots of reasons that might not work for you.

3. You could make a proposal for a Jenkins project. Again, that may be something you just don't want to do, for technical, business of political reasons. But creating the proposal isn't difficult and again plenty of people to help. I'm sue the proposal would be treated equally, and presumably you wouldn't have to do as much heavy lifting for the IP issues. Of course, you might be encouraged to try to combine projects, perhaps under an overall CI umbrella. I'm just speculating here, not trying to create extra headaches for the foundation.

To some of Andrew's specific concerns. "since the split to corporate-sponsored work as compared to the more independent developer-driven Jenkins". Really, the one thing I'd say about Eclipse governance is that it is developer driven. There are some very important caveats about that, but it is really true in the day to day work.

Now to whether Hudson is Java driven or not.. It really doesn't matter. As project contributors even the project leads can't tell you *what* to work on. If you've made a commitment on something of course it would be assumed that you'd see that through. But it really is a vote with your feet/code thing. If a group of folks wants to work on something the project lead can't stop them. Worst case the lead could refuse to include it, but they'd have to have a really strong justification for it. A bigger issue is of course when there are conflicting design agendas. I think that that is the thing to thing through carefully. Are there any insurmountable design differences, i.e. that couldn't be solved by abstraction. The second point is that Eclipse isn't really as Java driven as you might think. Besides of course supporting all kinds of different languages, there are a number of efforts based on other platforms, notably Orion.

Anyway, some thoughts, hope they are helpful.

I can't help but speak personally for a minute about my own spiritual tradition and why I'd care about this, seeing as it isn't something that really affects me directly. As a Buddhist -- whatever that means -- I don't see conflict as a problem, but as a precious opportunity. It's not something to avoid or even to try to fix, but a part of the path. Conflict is an opportunity to bring our compassion and mindfulness -- however meager -- from the realm of the theoretical into activity that can actually benefit people. All of us -- and it doesn't matter what we call our own approach to things -- can benefit from seeing conflict this way. When people disagree with us or cause us harm, our habit is to push back. When we decide not to go with those habits but instead do something completely unexpected -- that's when we start having fun!

"Victory over Conflict" is another way to say "Victory over War" which is the slogan of the Dorje Kasung. Members of that organization have written an excellent translation of the Art of War.

Hudson, Jenkins and Eclipse

noreply@blogger.com (Miles Parker) — Wed, 04 May 2011 18:17:00 +0000

Wow. Am I the only one who is hugely ambivalent about the Eclipse Hudson announcement? If William Safire were still with us he would point out here that Ambivalence does not mean "unsure", "undecided" or even "conflicted"; rather, it means that one holds two opposing -- often strongly opposing -- views at the same time. I have strong views, but I don't have a dog in the hunt at all, except that like many of you I rely on Jenkins and Hudson to actually build my software. And as there is big money and big corporate ego involved here, I should probably keep those views to myself. But naturally, I won't.

While the news isn't completely surprising -- there were some odd hints here and there -- it still raises a number of interesting and even challenging questions for our community. My guess is that this is in a way a harbinger of deeper changes in the overall role and mission of Eclipse and so I think it is worth talking about all of that openly.

I [Valence +] am thrilled because this is a huge win for the Eclipse Community. Transparency, good governance, common communication and tool infrastructure -- above all, a trusted community and process -- that's what Eclipse has to offer to open source projects everywhere. The fact that Oracle has chosen Eclipse as in effect the trusted repository for all of their CI related IP, community, branding and other assets speaks really well of Mike and the community as a whole. Basically, Oracle fucked up big time. Now they're making an attempt to do the right thing. Does it matter why, as long as everyone knows the emperor isn't actually wearing any clothes? And it is also yet more evidence that companies are going to be forced to move to a truly shared open governance model. The days when companies could get away with open sourcing in name only are numbered. All of this is great news for the open source movement. Oh, and speaking as an Eclipse project lead, the prospect of having an Eclipse project devoted to builds and integration makes me happy indeed.

I [valence -] am worried because it feels like a big loss for the Eclipse Community. The Jenkins / Hudson split was probably the most dramatic open source news of the year, and personally I loved every minute of it. If the Hudson move to Eclipse demonstrates the power of open source, Jenkins embodies its spirit. The name change was spectacular judo and -- to mix martial arts -- better than watching Kung Fu Panda. Deeper still, the name change puts out a real challenge to the corporate mindset, one that hasn't been really appreciated. That's for another post, but I believe the next big wave for open source is the triumph of community over branding, and substance over labels. Now, to me the deepest Eclipse principle is that actual contributors lead the project. If you didn't write the code, you don't get to make the decisions, unless you're actually my boss. Fundamentally, the Jenkins folks left Hudson because Oracle was violating that principle. I'm also concerned because the Hudson proposal puts all of us -- including independent committers like me -- in the uncomfortable position of being on a "side" of a battle that we didn't choose. And we wouldn't even be talking about this if the Jenkins folks -- i.e. the people who actually wrote most of the code -- hadn't had the courage of their convictions.

There is the smaller issue of how Hudson fits in to the Eclipse project ecosystem. Every other project that I can think of has some connection to the same basic OSGi architecture. Is this the beginning of a redefinition of what Eclipse itself is? I think Orion is pointing that way as well, and I don't think that that is necessarily a bad thing, but it is worth discussing.

So where does this leave us? I don't know. Even if I fell down on the [Valence -] side more, I still wouldn't actually oppose the Eclipse project, and it wouldn't matter a bit if I did. Ultimately projects need to sink or swim on their own merits, and the fact that Hudson will be subject to the same rules as everyone else makes me feel ok about that. But the process should not be pro forma either and should genuinely involve the whole community, especially the Jenkins folks if they chose to get involved. For me the best outcome would be for the Jenkins team to be invited in as co-equals. I wouldn't be a bit ambivalent about that.

Meet Butterflyzer Part II: What does it do and do you care?

noreply@blogger.com (Miles Parker) — Tue, 03 May 2011 19:19:00 +0000

Well.. glad you asked. I think. In my last post I shared what motivated me to create Butterflyzer. In this post I wanted to talk about why you might want to give it a closer look. What does this thing do anyway?

So, you want an Elevator Speech?

First I have to take one of my patented (ahem) detours into contrariness. There is a lot of conventional wisdom out there about how a product should be developed, marketed and, most importantly, capitalized. (I was thinking about "butterflyzer", but the lower case thing has been done to death.)

The "elevator speech" is a deeply ingrained part of Silicon Valley mythology. Here's the idea: You're riding a random elevator somewhere off the 101, and you happen to notice that standing right next to you is a Kleiner Perkins VP and he or she happens to be not quite on his or her iPhone. So you whip out that succinct but compelling paragraph that you've been fine-tuning and rehearsing for months. Two hours later you're sitting around a solid mahogany conference table with a bunch of your closest new friends, Montblanc in hand. Come on, we've all had that dream at some time or other. The one that turns into a nightmare when you discover that you're that one guy in The Social Network who apparently didn't bother to actually read the shareholder agreement.

It's true that good ideas are often easy to explain. But is the inverse always true? As an exercise, try coming up with a elevator speech convincing someone to invest in VisiCalc before anyone had ever seen a spreadsheet. Or a web browser in the mid-eighties. I'm not saying that it can't be done (probably a B-school exercise, come to think of it) but it's going to lose a lot in the condensing and the part that isn't going to come out is the overall deeper vision -- the thing that makes an iPod not just another MP3 player. I'm not that observant a guy, but one thing I've noticed about Silicon Valley is that a lot of the buildings there are only two or three stories high. Now, look at all of the innovations over the last ten years or so compared to the innovations in the previous ten or the ten before that. Is it any coincidence that these innovations have become progressively more feeble, more one-dimensional? They make a lot of money and everyone talks about them, but Facebook? Really? The whole thing makes me think of pop music. Does it have a snappy hook? Anyway, as you've probably figured out by now, the previous two paragraphs were just a cheap excuse for why I don't have a decent elevator speech. But I'll give you my bad one, since you (didn't) ask...

"[bing] Um, ok, so it's an application that... Well no, it's not web based, actually, although we can make one of those, it just won't be as fully featured, but the product really benefits from a rich client experience...anyway, like I was saying it's integrated with a web browser...yes, yes, I know that there are already a few companies -- yes, like Apple and Microsoft -- making web browsers and right, yeah, they give them away for free, but [bing] that's not relevant, here's the cool part.. Oh, it is? OK, nice talk.. [footsteps recede, doors close] ..ok, bye. [under breath] Nice Porsche, [bleep]."

The good news is that the last thing I want is traditional venture capital. Yuck. But that doesn't mean that I don't want to talk with people that have management expertise, money or other important stuff to contribute. I just want to be sure that folks understand that vision is something to share, not to negotiate.

Oh hey, I just glanced over at the right-hand column there. Maybe that's useful.

"Butterflyzer™: a revolutionary tool for researching, analyzing and visualizing social media."

Sounds cool -- if a touch hyperbolic -- but it doesn't help much, does it? So how about a long-form elevator speech? Think Petronas Towers and then the power goes out. And you forgot to charge your iPhone.

Butterflyzer Features

Here's the manifesto, i.e. the part you should have skipped ahead to if you were looking for so-called "content". The idea behind Butterflyzer is to provide an integrated set of tools that lets you:

Do all of the things that you currently do on the web, but in a much more organized way. And it's fun!
Do a bunch of things that ordinarily you would need to hire or become a web analyst to accomplish. Javscript, JSON, REST, SOA, etc..
Navigate and comprehend connections in the web in natural but completely new ways.
Analyze those connections using powerful tools and techniques previously only available in much higher-end tools, if at all.
Easily share the results of your explorations with others.

And all of that functionality should probably cost you around $100. Of course, there will be the inevitable Enterprise editions, etc.. but all of the cool stuff will be in the base version. We've got two main slogans that capture what is really unique about Butterflyzer:

Think outside of the Browser.™ If we step back in time a bit, browsers really haven't come that far since Mosaic. That's because they're stuck thinking about better ways to present single web pages. I can't reveal everything that supports this, but a lot of the magic is in the current Alpha version. The basic thrust is to think about social media browsing as a holistic, shared experience.
See the web as it was meant to be seen.™ Hey, it's a web, right? As in a network or a graph? Aren't there tools for doing that sort of thing? Well yeah, there are, and while it seems like a neat thing to do, there haven't been any mainstream tools that do it all that well. Why? Is it because most people simply can't handle the idea of a graph? I don't think so. Sparse graphs like we see in business process diagrams are easy. But it's actually quite difficult to make visual tools that fluidly, naturally and automatically support interaction with large interconnected graphs. I think we've made a lot of progress in this area -- in turn based on a lot of great work by other people in the Eclipse community -- and there is a lot more potential to exploit.

Let's get a little deeper into how this is done. There are many aspects to how the tool provides all of this. We could think of this as a sort of web surfing life-cycle but that's really just techno-babble, and the way you'll actually work with Butterflyzer isn't this linear.

Browse

You surf using the integrated browser. While Alpha users will notice that the usability isn't quite up to the level of current commercial browsers, we'll get there easily. The browser itself uses the same web engine as the one you're using now -- in fact you can even pick the engine you want to use. So everything renders quickly and looks as nice as in your standard browser. (It's also very easy to import the things that you find using Butterflyzer back into your desktop browser.)

Search and Collect

While you're browsing, pages are collected seamlessly in a number of different ways. For example, you can click a button to collect pages and links from sites you visit. (We can also do this automatically but that might be overwhelming.) Most importantly, in addition to the page information itself, you get all of the other information that you get with searches as described below.

Right now, when you do an internet search the information is provided to you from one service in a way peculiar to the service you're using. You can't really get information from multiple places, least of all when the information represents completely different kinds of information. Here's what you get when you do a single Butterflyzer search using the current default settings, with more to come:

Google Web Searches You know them, you love them.
Topsy Searches Topsy provides current pages and tweets that reflect what people are currently interested in.
Topsy Analytics All results are tagged with Topsy rankings and other information so you know what other people are finding valuable.
Live Tweets What are people saying about the topic now?
Content Authors Who is saying it?
Yahoo Placemaker Geo-location for pages and tweets where appropriate.
OpenCalais Semantic tagging for all of this, so you can quickly identify which information you're actually interested in.
Twitter Users This is all integrated with a deep set of features for Twitter Author and Tweet exploration.

Catalog

It is easy to collect a lot of information from the web, but what do you do with it? Think about how clumsy the process of simply creating and managing bookmarks is using conventional browsers. Butterflyzer also allows you to organize items in a tree, but these aren't simple bookmark hierarchies -- they're live connections to the data we've collected above and they can be organized in much more sophisticated and dynamic ways. But the really cool thing is that thanks to OpenCalais, cataloging is done automatically for you! As a really simple example, when you search for a page on Obama, you'll get an entry "President Obama" under "People", "White House" under "Organization", etc.. You might then create a "Politics" collection, perhaps as part of a "Interests" collection, and so on.

And as anyone who has ever "collected" old magazines knows, it's a lot harder knowing what to throw away than knowing what to keep. Butterflyzer provides powerful but easy to use tools for rapidly pruning away all of the dross and clutter. When you delete any of your pages, they're automatically removed from every collection. (Seems simple, but bookmarks don't work that way!) You can create boolean categories for your collections that allow you to easily combine and restrict member pages.

Curate

Cataloging naturally leads into curation. You have the information that you want, but do you know what it contains and why it might be important? A lot of the curation process can be accomplished simply through working with collections, but in addition to current support for keeping track of things we're building in rich meta-data features. Most significantly, we're working on support for page-scraping and other techniques to collect this meta-data automatically.

One of most important aspect of curation is integration of various data sources, and this is a key Butterflyzer advantage. As we described above, when you search and collect pages, you're actually getting information from a number of (properly credited, naturally) services and it is all seamlessly weaved together. For example, when you look at a web page, you can quickly navigate to all of the related tweets. Of course, you can get all of this information from various web tools, but think of how much time you spend just moving from one search page to another.

Navigate

Now we get to the fun part. How many tabs do you have open in your browser right now? (For some reason, I can't stand the things, but a lot of people use them.) How do you find things that you're looking for? When you think you're done with something you close the window. Then you have to go and do that Google search again. Or did you find it in Topsy? And once you're done all of the work you close out of your browser and it's gone. Unless you can find it buried in your browser history... somewhere.

How can we make the navigation process more comprehensive, manageable and timely? Well, we need to exploit all of the information we've been able to collect above and incorporate it back into the browsing experience. And then we need to present it to you in a way that you can see what you're looking for and everything it is related to at a glance. That's what the Butterflyzer graph tools provide. But the graph tools are much more than a way to find the information you've already collected..

Explore

The graph tools allow you to get a sense for the bigger picture. How are all of the things that you've been looking at related? What kind of information is on this webpage that isn't available on that one? Who's got something to say about this, and how are they related to other people? It would be nice to be able to see this information without visiting each individual page, wouldn't it?

But Butterflyzer's deeper agenda is to find stuff you weren't looking for but that you're really glad you found. If Butterflyzer can do that well, then we'll have accomplished what we've set out to do. After all, you probably can find everything you need using Google. It might take time, and you might have to dig through a bunch of irrelevant stuff, but you'll find it eventually.

What you won't be able to see from the search results is the deeply embedded information linked to your local information context. That's because Google doesn't really know much about your local context -- beside what it is presumably inferring from your previous searches. And sorry, but generic semantic web tools really aren't going to get you what everyone thinks they will, for all of the reasons that I discussed in the last post. But because Butterflyzer does know -- not in a sneaky way, but because you've actually told it -- is what you're interested in right now and what you were interested in a while ago within the same context. This sort of seamless local contextualization is the key to what we might call the "Butterflyzer Effect" (sorry) and it is simply not possible without close integration with the browser itself.

Analyze

Now, we want to get deeper into what we've collected, curated and explored. We want to understand how things are related in a more rigorous, reportable way. We can do that with the graph analysis tools and we'll be developing this capability in a lot of novel ways. But what you can do now is already quite powerful.

For example, let's say that you are an Oracle executive and you want to find out how recent changes in OpenJDK are being received. You could do a search through google reading through random sites and then do a Twitter search to see who is talking about what. Or, you could use Butterflyzer to do things in a much more systematic way. You could search for other companies and see how their OpenJDK related news is being received. And you can correlate all of this information.

You can easily collect and integrate charts on interest in various subjects (something that search providers seem to deliberately make difficult). And -- this is a really industrial strength feature -- you can then integrate them with tweets, news items and other sources to create a single timeline for events and interests.

Now that you've collected, curated, explored and analyzed this information, what else can you do with it? Whatever you want. Buterflyzer information isn't kept on someone else's server somewhere. Of course, your usage is governed by the terms of service from the original provider, but we don't tell you what you can do with it. Instead, we help you to share it with the rest of the World Wide Web.

All Butterfyzer data is stored using OMG standards based technology (EMF) that allows us to swap out the back-end, which will allow us to create future versions to support all sorts of storage and communication approaches, including all of the common RDBMs systems, fully collaborative multi-user environments, and even really advanced stuff like MongoDB and High Replication Datastore. (If you don't have any idea what I'm talking about, this is all just internal plumbing.) Currently we support two file storage methods; an easily parsed XML and a highly efficient binary format. Butterflyzer files can be emailed or shared on servers. (You could even use a tool like git to keep track of shared versions.) Butterflyzer information can be exported to bookmarks files and we'll be developing export features for many other common formats.

But perhaps the neatest thing that you can do with Butterflyzer is to create web documents that you can then provide to others. With a couple of mouse clicks you can build faceted search pages with tables and indexes, interactive timelines and even interactive charts with annotated web links. All of these are based on open web tools and can be shared without any restriction. It's your data, right?

What next?

This part is up to you. We're taking a novel approach to measuring interest in and potential for our product. If you like any of what you hear, and you want to see Butterflyzer development continue, then you need to let us know. Getting involved in the growing Butterflyzer user community is simple. Download the free alpha and follow us on Twitter.

One other note for the business and media sides of the technology house. We'd love to think we can bootstrap the entire thing so that we'll have the freedom to innovate in exactly the way we want to. But that may not be realistic or even desirable. Satire aside, capital and, just as importantly, an experienced management team are a critical part of the picture. Perhaps you have resources or executive experience that might multiply our potential, and if what I've said so far makes sense to you, let's talk! And if you're a media person, market analyst or blogger, I'd love to hear from you too. I think we have an interesting story to tell.

Meet Butterflyzer Part I: Is this any way to develop a product?

noreply@blogger.com (Miles Parker) — Wed, 20 Apr 2011 00:10:00 +0000

It's been a while since I posted and I've got quite a backlog of things I'd like to write about.. EclipseCon 2011, Agent Modeling Platform improvements, my mixed feelings about Big Data and AI, and so on. But what I've really been excited about recently is the progress being made on a completely different piece of software, and I'd like to share it with you. It's called "Butterflyzer".

As a bit of background, while I haven't blogged much about this, I've done a good deal of work in two different fields outside of my general focus on creating artificial models of real world complex systems. Along the way, I've invested a lot of time in learning how to leverage various Eclipse-based technologies to do the things that I really want to do. (For those who aren't involved in software development, Eclipse is the leading Open Source development environment and a growing platform for general purpose application development.) All of this has led me to a set of somewhat contrarian and untested views. So I thought.. hey, if not many other people share these ideas, they either a) not very good ideas, or b) an opportunity. As usual, I've saved time by simply guessing b) and then spent an obsessive amount of time and energy in creating an application that attempts to support that guess. Here is my experience, for what it's worth.

1) I spent a few years doing fascinating research on fine-grained agents for reasoning. I'd been aware of the theoretical limitations of AI, but that work exposed me to the real-world limitations of knowledge representation and machine reasoning. That experience has left me deeply skeptical about the sound and fury surrounding efforts like IBM's Watson, Wolfram Alpha and Hadoop. I'm not questioning that what they offer is valuable and interesting, but I do wonder about how they're being presented and understood, especially by the broader public. There are important -- even profound -- issues that are simply being glossed over. As I say, that's a subject for another post. But what that work did convince me of was the value of light-weight, "human-in-the-loop" approaches to using machines to support reasoning. I came away from it with the feeling that we should let computers do what computers do well -- sift through lots of data discovering patterns that fit existing templates, and let people do what they do best -- invent new templates. My guiding passion is to develop tools that harness the power of the computer to enable people to think more deeply and openly about our world, to share that information, and to make wiser choices. While the evergreen challenge for AI seems to be creating software that comes up with good answers to hard questions, I'm more challenged by the idea of creating software that comes up with good questions that have no answers.

2) More recently, I've somewhat accidentally spent a lot of time on Social Media research. Again, that work has led me to the conclusion that most of the efforts at Internet research have been too focussed on simple answers to frankly pretty lame questions. "How can I increase my click-through rate?" (Gee, I dunno, say something actually useful or at least interesting?) "What's the best day to post on Twitter to get maximum RTs?" (Hmm.. I guess "interesting" isn't really an important metric, judging from Twitter trending topics.) That might sound hyper-critical or even arrogant. Don't get me wrong, there are a lot of really cool efforts out there. But most of them are self-funded or academic exercises. I'm talking about where the money is, and as usual, that's in telling the cleanest story you can to investors. For that you need a metric and -- since that metric usually can't be money, since there often isn't any -- it has to be some other single dimension, like, yep, click-through rates and retweet counts. Now, I don't know about you, but I can't think of anything more trite and less worthwhile than trying to optimize that kind of stuff. So at base my motivation isn't arrogance, it's frustration -- we have all of these amazing technologies, and we can do so much better.

3) I've spent a lot of time developing desktop applications and working with other people that do the same thing. Since we have a good hammer (tools for building "rich" clients, aka applications) I'm certain that that it's the right tool to use. But pretty much the rest of the world is convinced that screw-drivers ("thin" clients, aka web pages and services) are the way to go. Everyone is moving to web-based, software as service applications. OK, they've long since moved there, and are now moving on to the cloud. Really, rich clients weren't even cool in 1998. Granted, some applications still demand the kind of high-performance, deep UI experience, and low-latency that you can only get from a desktop app. But just about the last thing that it makes sense to use a desktop application for is web analysis. Who'd be dumb enough to do that?

So, to cut to the chase, there are a lot of tools out there that:

1) Analyze massive biga-bytes of data and come up with answers to questions that no one ever thought of asking before. As Mike Olson, the CEO of Cloudera, one of the big players in the field, said "The light goes on.. Ah! I'm going to use a 1000 servers, because I can". Which means that people don't end up asking this critical meta-question: "is this question worth asking?" Still, given the number of questions being asked, it's inevitable that some small subset of questions are worthwhile. Undoubtedly, some of the information that Big Data provides us with is really useful.

2) Take that information and present it to end-users in a succinct, "actionable" way. I've spent a lot of time analyzing the sort of data that everyone else seems to be analyzing, and what I discovered was that, um... it was surprisingly hard to say anything interesting about any of it. That's my fault. As anyone who has bought breakfast cereal or red wine can tell you, packaging beats content every day, and I'm lousy at packaging. For example, I studied the events in the Middle East using the same kind of data that Sysomos reported on in this blog. Cool, huh? Now, this is how I reported the same data to my colleagues and clients: a) We don't really know where any of the Tweets are coming from, b) There were more tweets after something happened then before anything happened, and c) People used terms like "people", "time", and "RT"*, all of which implies that d) we've been wasting money and time on this and we should change our approach. That just doesn't come off as well. I should have made more charts and graphs. I don't mean that in a totally cynical way. To me, interesting means "surprising" and "novel", but I'm happy with a definition that includes "pretty" and "compelling". Visualization allows us to really get our minds around information, but that information needs to be meaningful in the first place.

3) Provide all of that data packaged as SaaS (web-based) analysis tools. These are described with words designed to make CIOs feel savvy when dropping wads of cash on said tools. "Business Intelligence" is a must. (Because oddly enough, no-one wants "Business Stupidity" tools.) "Integrated" and "Solutions" are always good filler. "Continuous", "Cloud", etc.. are still pretty hip. "Big Data" is totally hip. Hey, what we really need is a Business Intelligence System for Automated Monitoring and Assimilation of Massive Industry Buzzwords. We could integrate that with our CMS and then attach that to our CRM for real-time customer sales feedback, and then... But I'm sure someone is already in this "space". OK, getting off into critique for the sheer joy of it, and that's not very helpful. But let me just point out that the one thing that you don't see on any of these websites is how much the product actually costs. Hey, if you have to ask, right? And look, these aren't products, they're services, ok, so why don't you just let us send you a bill in a couple of months? As I write this, that kind of approach is sounding pretty damn smart...

Snotty criticism aside, you can't argue with success. And people have done some really ground-breaking and impressive work along the way. It's certainly not as if the limitations of these technologies aren't well known to the people actually working on them, and if other people are willing to spend money on technology they don't understand, then that's fine. Full employment for software developers, computer scientists and mathematicians is a good thing, probably. Keeps us off the streets.

But the thing is, I don't want to create tools for CIO budgets, I want to create tools that people will actually use, be delighted with and do interesting, unexpected things with. And (as demonstrated in the screenshots above) easily share their discoveries with others without having to pay a tax on the results. This requires software that 1) doesn't need unlimited infrastructure and data sets, 2) takes an open-ended, exploratory approach 3) has a responsive, engaging UI, and that you can download, install and use on your own computer for whatever and however long you want to. And it probably needs to cost less than a hundred bucks. I think it's worth trying. The attempt is called Butterflyzer.

You can download the free Butterflyzer Alpha and see more screenshots and screencasts at http://butterflyzer.com. (Psst.., the first 50 people who follow us will get 50% off the retail price when and if we release a commercial product.) I'd love to hear what you think!

To find out what Butterflyzer actually does, check out Part II.

*Actually, there are some interesting things you can learn from such analysis, and I'm not really doing the Sysomos graph tools justice. Here's Butterflyzer's take on similar but more recent data. Here, we've integrated actual sample Tweets. Things are a little noisy graphically but the tools allow you to easily narrow and expand the information shown.

Sex, Lies and Eclipse

noreply@blogger.com (Miles Parker) — Wed, 23 Mar 2011 03:17:00 +0000

Now that I've stolen your attention with the oldest marketing trick in the world, you might as well check out my EclipseCon talk Wednesday at 2:50 in Stevens Creek: "Get Real? What is Eclipse AMP and why should you care?"

I can promise you that a) I will talk about sex, and b) it won't be the least bit prurient, inappropriate or even mildly titillating. Sorry to disappoint. I am also going to talk about lies and perhaps tell a few of my own. And yep, I'll even talk about Eclipse. So, what do Egypt, Watson, Teenage Pregnancy, Big Data, Nuclear Reactors and the Eclipse Modeling Framework have to with practical stuff that you care about? You're just going to have to have to come to the talk and sort the truth from the fiction for yourself.

Supporting Multiple Resource types with EMF Editor

noreply@blogger.com (Miles Parker) — Sat, 29 Jan 2011 23:13:00 +0000

The resource model is one of those aspects of the Eclipse Modeling Framework that is extremely powerful yet can seem way too complicated for us mere mortals to muck with. "But what if I break something?" The thing about powerful abstractions is that the more powerful they are, the more they do; and the more they do, the more systems they touch and -- somehow -- the more mysterious they become. I'll propose a little law here which I am sure is not original: "any abstraction that is sufficiently powerful will be initially opaque". Just where are those levers we need to pull to get things to do what we want? As a highly leveraged framework, EMF is full of such abstractions. So how to get past that learning barrier?

Well, it takes some up front commitment, really, but there is a pretty simple heuristic to it: Decide the things you want to change, look through the generated code for where the things are that you might want to change, and experiment. Or wait for someone else to do the experimenting for you, which is the smartest strategy of all. Ergo, you must be smart, because you're reading this. So without further ado, let's take a look at how to do something extremely cool.

The basic scenario is this: there are a number of different data formats -- in EMF speak, "Resource Implementations" that might be appropriate for a given user and use case. By far the best example of this is dealing with the problem of data transparency vs. data efficiency. XML is (arguably) readable using any text editor, and it is definitely the format you want to be using when something goes wrong, for example a reference is bad or you have a problem with encoding. But memory footprint and load and save performance wise, it's a total dog. In contrast, Binary is the format you want to use when you have a lot of data and you need to use it, but good luck figuring out what's in it without your editor.

Here's something that I'll bet many people -- even those who have been using EMF for a long time -- don't know. I didn't. EMF has built in support for a very efficient Binary Resource model. But how can we use it? There is (not last time I checked anyway) no resource type entry in .genmodel for "Binary".

In fact, it turns out that it is ridiculously easy to replace the generated XML resource support with support for binary resources if you're willing to touch a bit of code. (I want to thank Ed Merks and Kenn Hussey for pointing me the way initially.) That's great, but now you have the problem of sticking the user with an unreadable binary format. Wouldn't it be cool to support both formats, so the user could take the same data and save it in the format most appropriate for his or her needs? It turns out that that's easy to do too, and it took me (much) less time to figure out then it took to write this blog post. And actually, I need to get on with it, because I'm working this weekend and I really don't want to be. So without even further ado, let's follow the heuristic I mentioned above.

First, we know that we need to support multiple resource types. How do we change that? Well, resources are setup in the foo.model plugin. In foo.util we can see the various factories. So we need to create two new pieces. (Actually, we don't have to do this, but it makes it a little easier to see what it happening.) So we could look at what the existing resource handlers are doing -- say FooResourceImpl -- and copy and modify it with the appropriate changes:

public class FooResourceImpl extends XMLResourceImpl {

    public FooResourceImpl(URI uri) {
        super(uri);
    }
}

Suggests that we probably want something like the below (although it's probably unnecesary):

public class FooBinaryResourceImpl extends BinaryResourceImpl {

    public FooBinaryResourceImpl(URI uri) {
        super(uri);
    }
}

I told you it was ridiculously simple. Similarly we want:

public class FooBinaryResourceFactoryImpl implements Resource.Factory {
    public Resource createResource(URI uri) {
        return new FooBinaryResourceImpl(uri);
    }
};

Now, we want to support two different kinds of files. We could do all sorts of funky stuff there, but let's keep it simple. We'll just use different file extensions. This part may seem just a bit mysterious, just because it involves a number of different artifact types. But first, we know that genmodel defines the file extensions. We can add one for the binary resources in the model section. Let's use "foo,foobin". We're only doing that for consistency because the real action is elsewhere. Since we're now dealing with editor stuff it makes sense to look there for what we want. And if we look at the FooModelWizard, we can see that there is a reference to _UI_FooEditorFilenameExtensions. That's the bit we want to change, and because genmodel won't write over .properties files, we'll need to edit it there:

[foo.editor/plugin.properties]
...
_UI_FooEditorFilenameExtensions = foo,foobin
...

We next need to tell the editor what to do with those binary files when it gets them. The natural place to do that is when we setup the editing domain for FooEditor, and we'll do it at the end of the method call:

/**
  * This sets up the editing domain for the model editor.  
  * 
  * @generated NOT
  */
 protected void initializeEditingDomain() {
...
  editingDomain.getResourceSet().getResourceFactoryRegistry().getExtensionToFactoryMap().put("foobin", new FooBinaryResourceFactoryImpl());
 }

OK, now the more challenging part. How to actually allow the user to select the file type? Again the obvious place to look is in new file wizard. We just need one change here, to tell the wizard what to do when the user selects the binary extension:

FooModelWizard
/**
* Do the work after everything is specified.
* 
* 
>>  * @generated NOT
*/
@Override
public boolean performFinish() {
...  
        // Create a resource set
        //
        ResourceSet resourceSet = new ResourceSetImpl();
>>resourceSet.getResourceFactoryRegistry().getExtensionToFactoryMap().put("foobin", new FooBinaryResourceFactoryImpl());
...
}

Now, for extra credit, how can we let the user change the file type once a file has been created? Well, we know the user picks the file type in the editor, so let's look there. And here we go..we'll show the entire method, but it is only the first 15 lines or so that have changed:

/**
*  
* 
* @generated NOT
*/
protected void doSaveAs(URI uri, IEditorInput editorInput) {
    ResourceSet resourceSet = editingDomain.getResourceSet();
    Resource resource = resourceSet.getResources().get(0);
    if (!uri.fileExtension().equals(resource.getURI().fileExtension())) {
        Resource oldResource = resource;
        if (uri.fileExtension().equals("foobin")) {
            resource = new FooBinaryResourceImpl(uri);
        } else if (uri.fileExtension().equals("foo")) {
            resource = new FooResourceImpl(uri);
        } else {
            throw new RuntimeException("Unexpected resource type: " + uri.fileExtension());
        }
        resource.getContents().addAll(oldResource.getContents());
        oldResource.getContents().clear();
        resourceSet.getResources().remove(oldResource);
        resourceSet.getResources().add(0, resource);
    }
    resource.setURI(uri);
    setInputWithNotify(editorInput);
    setPartName(editorInput.getName());
    IProgressMonitor progressMonitor = getActionBars().getStatusLineManager() != null ? getActionBars().getStatusLineManager().getProgressMonitor()
: new NullProgressMonitor();
    doSave(progressMonitor);
}

All we are doing there is creating the new resource, moving the contents over to it and replacing the old resource with the new one. (Don't be afraid to muck with resources and resource sets, they aren't as fragile as you may think.) That's it. I'm sure I missed something here that I'll discover later, but everything works.

Back to the original use case. I needed to support the xml format because I expect people to use the data in the application in a lot of different ways. But I really needed the binary format because my application produces very large files and those files take a long time to load. (When you load these files in the editor, the UI is frozen.) One file took:

19 m, 57 s

Using the binary format improved that a little bit:

00 m, 00 s, 419 milliseconds

Part of that is due to the fact that reference resolution can be deferred, but the entire file can be easily navigated without perceptible delay. The whole user experience is dramatically altered. That's a lot of functionality to gain with really some pretty small changes! And that's why taking the time to grasp more complex abstractions is worth the effort. What's interesting is that once you understand them, they don't seem at all complex!

EMF and RAP and Virgo, oh my..

noreply@blogger.com (Miles Parker) — Sat, 01 Jan 2011 05:28:00 +0000

Many of us who've spent time traipsing through the Eclipse wilderness are probably familiar with this story...

Tis a beautiful, sunlit 'morn and you're wandering around the world wide web when you come across a bright shiny page that says something like "Eclipse ABC does [exactly what you want to do] and [as we've worked out all of the old bugs] it is [now] [relatively] [unless you're using..] easy to integrate with XYZ". There is even a set of step-by-step directions to get your from point ABC to point XYZ. Now, even though you've been down this road many times before, hope springs internal in a [not so much anymore] young man's [or woman's] breast and you're up for a little adventure, as long as you come back before lunchtime. And in fact, the first part of your journey is pleasant, idyllic even. But then..

"..hmm.. this barren outcropping isn't on the map. And when did this tree trunk fall across the path? I think I already missed lunch. Still have to find a way around that somehow...maybe through this swamp over there. No, that definitely wasn't the way to go, let's back-track a bit. In fact, maybe it's time to head back, it's getting late. And what, waste all of the time I've already spent exploring this area? Let's climb up over this talus slope and see if we can get a bit of a perspective on things. CRACK!! ZZZTTT!! Great..almost zapped by a bolt of lightning and now I'm caught in a thunderstorm...typical."

So, you've struggled home at nine pm, wet, cold, tired and grumpy, and even though you missed dinner as well, your significant other greets you brightly. "Well, how was your day, dear?"

And somehow, your will is not broken. You've learned to accept this kind of thing as a natural part of the life of an explorer living on the edge. [OK, laying it on a bit thick here..] In fact, the very next day, you decide that there has to be a better way. Naturally, this being the land [ecosystem, if you prefer] of Eclipse, there is a better way -- or at least that's what the project page for a new wonder project subtly suggests. So you naively and dutifully follow their map, and head out for another morning of exploring, promising yourself that this time you'll definitely return before noon whether you've found the end of the rainbow or not. And...guess what? You walk out through the forest about 100 meters, take a left and then another right, push a few stray logs out of the way, and suddenly a clearing opens up and a beautiful maiden [um, what's gender neutral for "maiden"?] is standing in the middle holding a... OK, let's stop there.

The point is that, let's face it, often things in the Eclipse ecosystem, and oh, just about every other software ecosystem I can think of, just don't always really work as advertised. Sometimes they work really well, but there are glitches. Other times they don't work at all. Unfortunately you don't discover that until you've wasted enough time that you are then willing to waste even more time trying to solve all kinds of bugs and issues that really should have been solved a long time ago. By someone else. That's why it's especially nice when something does work. Exactly as advertised. The first time you try it.

I've been experimenting with solutions to putting EMF models on the web for a year or two now off and on. There are two really cool efforts out there, one based on RAP and the other based on GWT, and both of which Kenn Hussey, Ed Merks and others have put a lot of effort into. After minor fits, starts and glitches -- most self-induced -- I've been able to get cool EMF editors working on the web with relatively minimal effort -- insanely minimal if you compare to the time it would take to do a custom EMF front-end. But wait, did I say "on the web"? I should have said, "on a web page, launched from Eclipse". For I have not really been able to get past the point of having a nice little web app show up in self-hosted mode and actually being able to deploy it -- in other words, so that other people could actually use it.

I've tried and tried Tomcat hosting, but I guess I'm too dumb or impatient for it. Inevitably I got lost in the vagaries of WAR file creation, missing dependencies, obscure console messages and bizarre configuration issues, culminating through the deep dark labyrinth of the Tomcat logging system. Then -- wisely exercising good judgment and ignoring sunk costs for once -- gave up. This last time though I had something that I really really wanted other people to be able to use, and some quality time to devote to it and determined to solve it. I was encouraged in this effort because people have really understood what needs to be done to make this process more straightforward. For example, there is Holger Staudacher's GSOC effort for RAP WAR products [Where is Tupac when you really need him?].

Unfortunately this wasn't the end to my pain. And judging from the messages on the posting it hasn't ended everyone else's either. This isn't to diminish Holger's effort at all. To the contrary, it is the fact that this is such a difficult and log-across-the-path thing to take on that makes efforts like this so admirable and important. If all integration challenges were easy, we wouldn't need to worry about integration. All of the issues have to do not with the tools themselves, but with the target for those tools and the complexity of the underlying mechanisms, none of which the team has any control over. This is similar to the Buckminster and B3 effort. Efforts are needed because it isn't like Tomcat and Jetty and servlet bridges and other endless dependency issues are going to go away anytime soon, and dragons [or at least logs-in-the-path] be there.

So after stumbling around a bit, I thought [I'm getting to the point, finally] why not take a look at this Virgo thing? After all, as the RAP website says, there are two choices: Embed a server in Equinox, or Embed Equinox in an existing servlet container. And when you think about it, does it really make sense to layer one dynamic runtime technology on top of another if you absolutely don't have to? Like, isn't that just asking for trouble?

As usual, the documentation looked reasonable if not simple, and had the characteristic Eclipse modesty: "Here are the steps we used to deploy RAP on Virgo. We know the process can be improved, but just to give you an idea of how it works, try the following:" So I followed the steps, stumbled once or twice and [in less time then it has taken me to write this blog post and perhaps for you to read it] I had the RAP demo app up and running! With the latest [1.4] version of RAP even.

OK, great, but I'd actually gotten that far with Tomcat before. What I just knew wouldn't work was getting the EMF RAP stuff to work with that. I didn't quite get all of the console stuff and what not, but I just thought to myself, "what if I just dropped the EMF bundles into the same place, modified the demo .plan file to include them, and.. No way that could possibly work, right? Not only did it work, but it actually worked before I thought it would. I was ready for endless rounds of editing, rebuilding, restarting, reloading, re.. But instead, I saved the changes to the .plan file and Equinox/Virgo automagically hot-loaded them and ran through the dependencies there and then. I cleaned a couple of things up, and then I navigated to the URL, and there it was. Seriously, I haven't been that impressed by a technology that "just worked" since the first time I built an EMF model and editor. I'll say it: The Virgo/RAP/EMF stack is going to be -- already is -- a kick-ass combination. Light-weight development, light-weight deployment, industrial strength, web-based MDSD.

And actually, the steps are even simpler than outlined in the docs above (but it's likely you'll want to refer back to the details there). Oh, one special note, I've had it with being a build weeny. What I'm doing below is dumb, dumb, dumb. No product builds, no ant, no Maven, nothing but you, Eclipse and your m 'n f'in, drag 'n drop'n mouse.

I'll assume you have a working model in Indigo EMF RAP. Note that there is currently a minor issue
Download Virgo Kernel and put it wherever the hell you damn well feel like.
Download Runtime RAP (I used 1.4) and drag the contents of plugins into [virgo]/repository/usr
Download The OSGi Autostart Jar and put it [virgo]/repository/usr.

Go to your copy of Eclipse and copy-drag the following stuff from plugins into the [virgo]/repository/usr :

org.eclipse.emf.rap.*
org.eclipse.emf.common
org.eclipse.emf.ecore
org.eclipse.emf.ecore.xmi
org.eclipse.emf.edit

Export your plugins to [virgo]/repository/usr

Create a file called whateverthehellyouwant.plan [OK, does the wold really need yet another dependency artifact type?] with something like the text below, obviously updating the version numbers when you come across this page in six months [when none of it will work anymore anyway], then copy it to [virgo]/pickup:

<?xml version="1.0" encoding="UTF-8"?>
<plan name="virgo-emf-rap-demo" version="1.4.0" scoped="false" atomic="false"
xmlns="http://www.springsource.org/schema/dm-server/plan"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springsource.org/schema/dm-server/plan 
http://www.springsource.org/schema/dm-server/plan/springsource-dm-server-plan.xsd">

<!-- present in virgo 
<artifact type="bundle" name="org.eclipse.equinox.common" version="[3.5.0, 3.7.0)" />
-->
<artifact type="bundle" name="org.eclipse.osgi.services" version="[3.2.100, 4.0.0)" />
<artifact type="bundle" name="org.eclipse.equinox.preferences" version="[3.3.0, 4.0.0)" />
<artifact type="bundle" name="org.eclipse.equinox.app" version="[1.3.1, 2.0.0)" />

<artifact type="bundle" name="org.eclipse.core.commands" version="[3.6.0, 3.7.0)" />
<artifact type="bundle" name="org.eclipse.core.jobs" version="[3.5.1, 4.0.0)" />
<artifact type="bundle" name="org.eclipse.core.contenttype" version="[3.4.100, 4.0.0)" />
<artifact type="bundle" name="org.eclipse.core.runtime" version="[3.6.0, 3.7.0)" />
<artifact type="bundle" name="org.eclipse.core.expressions" version="[3.4.200, 4.0.0)" />

<artifact type="bundle" name="com.ibm.icu.base" version="[4.2.1, 5.0.0)" />
<artifact type="bundle" name="org.eclipse.help" version="[3.5.0, 4.0.0)" />

<artifact type="bundle" name="org.mortbay.jetty.util" version="[6.1.23, 7.0.0)" />
<artifact type="bundle" name="org.mortbay.jetty.server" version="[6.1.23, 7.0.0)" />

<artifact type="bundle" name="org.eclipse.equinox.registry" version="[3.5.0, 4.0.0)" />
<artifact type="bundle" name="org.eclipse.equinox.http.registry" version="[1.1.0, 2.0.0)" />
<artifact type="bundle" name="org.eclipse.equinox.http.servlet" version="[1.1.0, 2.0.0)" />
<artifact type="bundle" name="org.eclipse.equinox.http.jetty" version="[2.0.0, 3.0.0)" />

<artifact type="bundle" name="org.eclipse.rap.rwt" version="[1.4.0, 2.0.0)" />
<artifact type="bundle" name="org.eclipse.rap.rwt.q07" version="[1.4.0, 2.0.0)" />
<artifact type="bundle" name="org.eclipse.rap.jface" version="[1.4.0, 2.0.0)" />
<artifact type="bundle" name="org.eclipse.rap.ui.workbench" version="[1.4.0, 2.0.0)" />
<artifact type="bundle" name="org.eclipse.rap.ui" version="[1.4.0, 2.0.0)" />
<artifact type="bundle" name="org.eclipse.rap.ui.views" version="[1.4.0, 2.0.0)" />

<!-- TAKES CARE OF SERVLET LAUNCH -->
<artifact type="bundle" name="org.equinoxosgi.core.autostart" version="[1.0.0, 2.0.0)" />

<artifact type="bundle" name="org.eclipse.emf.common" version="[2.6.0, 3.0.0)"/>
<artifact type="bundle" name="org.eclipse.emf.ecore" version="[2.7.0, 3.0.0)"/>
<artifact type="bundle" name="org.eclipse.emf.edit" version="[2.7.0, 3.0.0)"/>
<artifact type="bundle" name="org.eclipse.emf.ecore.xmi" version="[2.6.0, 3.0.0)"/>
<artifact type="bundle" name="org.eclipse.emf.rap.common.ui" version="[2.7.0, 3.0.0)"/>
<artifact type="bundle" name="org.eclipse.emf.rap.edit.ui" version="[2.7.0, 3.0.0)"/>

<!-- YOUR EMF RAP BUNDLES GO HERE!! -->

</plan>

Got to command line and type:

cd [virgo]
export JAVA_OPTS=-Dorg.ogsi.service.http.port=10081
[Mac users:] export JAVA_HOME=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
bin/.startup.sh

Go to your browser and paste in: http://localhost:10081/thesamethingthattheworkedineclipse

That's it. Hope you find your way through the forest as easily as I did. If you don't, then just remember that you're probably not alone, the world is full of scary places, and even when something doesn't make any sense there is probably still something to learn. Take a listen to what Tupac is saying:

“I know it seems hard sometimes but remember one thing. Through every dark night, theres a bright day after that. So no matter how hard it get, stick your chest out, keep ya head up.... and handle it.”

Come to think of it, Tupac is probably not the best source for words to live by. So perhaps better to hear how "T Bone" Stankus does it:

I said, "Hey! Hey, weird little whiners, I am on a quest To dream the impossible dream. Walking down the road one day, doo-dah, doo-dah, I said, "Hey kids, I'm looking for the truth of life. Where do I go, who do I see?" They said, "Slow down, mister, in order to find the truth of life, one must see THE WIZARD!" I said, "THE WIZARD? Well, where does this wizard, old wise one, live?" They said, "You see the big, green, glow-in-the-dark house up on the hill?" I said, "Yes, I see the big, green, glow-in-the-dark house up on the hill. There's a big, dark forest between me and the big, green, glow-in-the-dark house up on the hill. And a little old lady on a Hoover vacuum cleaner going 'I'll get you, my little pretty, and your little dog, Toto, too!'. I don't even have a little dog, Toto."

Is GPL damaging to your ecosystem's health?

noreply@blogger.com (Miles Parker) — Thu, 11 Nov 2010 00:47:00 +0000

In my post earlier this week I sighed about the incompatibilities between GPL and EPL, took the FSF to task for their (in my eyes anyway) disingenuous posture and suggested a way to hack ourselves free of what (I perceive to be) the damage that GPL can cause. But as I was writing that post, I realized that I had things backwards. I talked about what bugs me about the FSF's attitude, and I talked about what I wanted to do about it, but I didn't explain why I cared. It was all about my negative motivation, really. Or perhaps my "negative positive" motivation -- to undermine that which one sees as a threat to getting what one wants. But I didn't talk about my "positive positive" motivation -- what I would actually like to see happen, or my "positive negative" motivation -- why I think there is a problem in the first place.

Eh, ignore all that, I'm beginning to sound like Donald Rumsfield. The point is...there are things I bitch about and that is generally because I want things to be different and I think I can make them so. This attitude happens to be the root of all neurosis, but oh well, here I go again..

What I want is to contribute to software ecosystems that allow the greatest possible resilience, diversity, and adaptation. To create that, one needs to have a large web of interaction. For example, the removal of one node should not cause a cascade of failure across all the dependent nodes -- we need substitutability. (As an aside, the bio-dynamic agriculture movement has a lot of very interesting things to say on these subjects.) To get this, one needs many different paths or edges between nodes. Now, for any ecosystem to work, one needs to exchange meaningful energy. That energy can take many different forms -- it can be food, money, solidarity, status, self-propogation, altruism or anything else that motivates human beings. And in my experience studying complex systems, it is those systems that support multiple forms of exchange that are the healthiest. In a sense (sorry, but I can't help but use the term here) one could say that these systems have a quality of meta-robustness.

What I think is that GPL harms the formation of these ecosystems more than it helps. That's a bold (though not unique) claim, and one that may not be right. But it is what motivated my prior post, and I thought it would be worth walking through some of the logic so that you can judge for yourself.

To examine this, let's look at the {x}PL landscape from the lens of interaction networks. We'll model the various ecosystems and see what pops out. I did this exercise in my head over the last year or so, and thought it was worth sharing. As always, there is no ultimate truth here, only a dynamic interaction or self-adapting and sometimes mutually correcting thoughts. So, it's impossible for my thinking to be "correct", but it is possible that it might be interesting or even helpful in some way to others.

First, let's get an idea of how this approach works, using a simple ecosystem that we are all familiar with. We’ll use the way-back machine and look at the dark days of pre-open source ecosystems. Believe it or not, such purely commercial ecosystems function to this very day.

In the diagram above, we can see a very simple relationship. Company 0 creates software and releases it. It may release an open API, in which case other companies can write to that API freely, with the expectation that customers will purchase both pieces of software, and/or it might license the software to Company 1, which might in turn resell that license to its customers along with its own. Of course, any other company may do the same thing. An extremely important -- and I think under-appreciated -- aspect of this system is that any other company can use the open API and make money off it. In particular, if Company 1 releases a closed library based on Company 0’s open API, Company k can license that closed library without needing to get permission from Company 0. And this is true all down the chain. We end up with an ecosystem that looks like this:

Of course, in this case the web is fed by money only, so it isn't ideal according to my criteria, but it does function pretty well on its own. If we look at the space of the graph, the vertical dimension loosely measures the breadth of functionality (the graphs for the other webs below don’t spread as far in this dimension, but don’t read anything into that, the important thing is that they fade into the distance). The horizontal dimension loosely measures the depth of the dependency chain. Note that this is an extremely important dimension, because it has a strong correlation with reuse and abstraction. In general terms it represents the level of complexity and leverage that independent entities can contribute to the ecosystem.

This purely commercial food web can spread out arbitrarily in either direction, but I propose that there are natural constraints that tend to limit horizontal growth. I’d like to build an Agent-Based Model to more formally test these theories when and if I get the time, but the basic intuition is simply that as we get further and further to the right, the APIs generally get more specialized. That is simply because the markets are always a subset (or intersection in the case of multiple dependencies) of the APIs that a given Company’s product is dependent on. As the market shrinks, the incentive to create software shrinks. This is all entirely consistent with what we see in the real world. There tend to be software creators at the left-most side (Microsoft, Oracle), various relatively large software makers that produce software for those systems (when they aren’t eaten up by their dependency daddies, a very frequent occurrence) to the immediate right (Adobe, WordPerfect, Blizzard), and scattered smaller companies that feed off these ecosystems (Photoshop plugin makers, VARs).

Got that? Now let’s look at another straightforward system. This is a pure open source system. Here we’re looking at GPL because GPL is explicitly designed to foster such a system, but actually you don’t need strong copyleft per se. This model would work as well for any open source system where there isn’t commercial activity.

This is actually even more straightforward, elegant even. Any (compatible!) license can make use of any other license. This gives rise to an Ecosystem that looks like this:

The food web can be quite deep and rich, and indeed we see exactly that in real world GPL ecosystems (Linux, R). GPL based systems can and do grow in both dimensions, creating very beautiful and holistic ecologies. But, we need to ask a question here. Where exactly is the food? We know some of it comes from personal enthusiasm/poor judgement (guilty), some comes from universities, and some comes from corporations with an interest in selling software. But note that in any case the energy exchange isn’t really well aligned to the network structure. We’ll see the implications of that below.

Now is probably a good time to reveal another bias of mine: you need money to produce good software, or at least to produce a significant amount of it in a form that end-users actually want to use. This may be a place where you and I disagree, and if so, fair enough -- you can stop reading, because the rest of my argument won’t be very convincing to you. [Insert smart-ass emacs comment here.]

Next, we come to a much more complex ecosystem, and my personal favorite. This is one that attempts a sort of voluntary shot-gun wedding (yes, there is some contradiction there) between the open source and commercial worlds. Here we’re using the example of EPL, but the license doesn’t have to be weak copyleft -- it could be an even more permissive license like BSD. And (I was going to say “of course” but "of course" it is not that simple) you can mix them all up.

Here, we have companies creating EPL code and at the same time creating code that leverages -- or in some cases is entirely independent of -- that EPL code. We immediately notice something here: there are a lot more arrows! Again, my intuition is that more arrows equals more diversity equals more potential but your intuition might be different. Anyway, if we look at the food web that creates, it looks just as rich, if more colorful, as the GPL web:

But notice that unlike the GPL web there is something more going on here. To the creative and autonomous energy of open source, we add the really practical energy of money. So the thing I love about this ecosystem is that not only can it grow in both directions, it has the energy to do so. Most importantly, unlike the commercial ecosystem it is not limited on the horizontal axis. Why? Because the market size doesn’t shrink as you go to the right. That is because to be commercially viable you don’t need to rely on vendor specific API trees. As the overall ecosystem is not based on or dependent on any particular technology, you can place yourself in the space where you can contribute to and align with any API/functionality subset. And because the backbone of the system is typically the open components (largely because social forces drive it there), if the APIs are too locked into a specific upstream dependency, you can change them!

As above, I argue that this is precisely what we see in real weak-copyleft/commercial ecosystems. Commercial vendors and others (IBM, WindRiver, Google) produce software that other software can interact with via license, API or open source (BIRT, Android SDK, XText, RAP, Blackberry) and these can work with each other. In fact, if I can make be chauvinistic for a second, perhaps this is why the Eclipse ecosystem is leading all sorts of very creative and market friendly initiatives that seem to feed off of one another in all sorts of different and unexpected way (Xtext->iPhone, SWT->RAP). I always know when I’m modeling a complex as opposed to boring system -- I see connections that I didn’t anticipate.

Note something else interesting with my examples above by the way. Technologies that are based around GPL are using EPL tools to create content. So that is a part of the weak and strong interactions that works well. Why shouldn't we try to to extend that success to the software itself?

Warning: now my argument turns critical. I was perhaps unduly harsh on GPL -- or really, the FSF -- in my earlier post, but for the most part, I’m really motivated by the desire to see healthy ecosystems. While I do have an interest in EPL -- I'm an Eclipse Committer after all -- I did not create this analysis to support my point of view. Instead, I came to my point of view based on this analysis. If there are faults in my analysis I agree to alter my point of view.

So then, let’s look at what happens when we mix GPL and commercial licenses. This final diagram is quite a bit more complex but there are really just a couple of key interactions.

Clearly the GPL and Commercial systems can co-exist just as in the prior diagrams, but only to the extent that they are completely separate from one another. And to that extent it isn’t a mixed ecosystem at all but merely two distinct ecosystems.

The EPL cannot exist in either part of this system. It can’t play in the GPL system of course. I’ve got a parochial interest in that issue and let’s just say that that leaves out a great chunk of software, but that really isn’t a problem we can pin on GPL alone. But remarkably, it can’t play in the commercial part of the system either, for reasons I’ll explain below, and we can pin that one on GPL.

So how do the GPL and commercial entities interact? This is the crux, because whenever the GPL is criticized for incompatibility with market incentives, people argue that the two worlds are entirely compatible. As Seweryn Niemiec suggested in response to my last post: “Publish your software on GPL and make money from it too.” True, as far it goes. The idea here is to “dual license” your code. I note with some irony that this stand goes against the spirit of the GPL -- or at least that of the strong strong copyleft adherents -- and in that sense itself represents a kind of self-help IP hack itself. But there is a much deeper, subtler, and far reaching issue here; one that I think deserves contemplation from anyone interested in understanding the complex issues involved. It rests on network dynamics.

Look at the diagram above again. Suppose that Company 1 wants to charge money for a license to their software. In other words, they simply want to play the same game that Company 0 is playing. Seems reasonable, but they can’t do it without obtaining rights from Company 0 and all that goes along with obtaining those rights, i.e. money, lawyers, non-competition, ad infinitum. Why is that? Because GPL terms prevent Company 1 from using the GPL code. So Company 1 needs a license to Company 0's commercial code. Well, can’t Company 1 write their software anyway, and get the user to buy a copy of Company 0's software and a copy of Company 1's software and install them together? According to the GPL license -- or at least the preferred FSF interpretation of it -- no! Here (my apologies for taking so long to get to the point) we find the real damage from the diversity point of view, and the justification for the IP hackery I describe in my post yesterday.

I want to route around that damage because the implications of the above are even deeper than they first appear. The sub-license problem is transitive. Again, looking at the diagram above. Company k does not just have to license from Company 1 -- they must license from Company 0 as well. In fact they must license from every company in their entire chain of dependencies. When multiple companies license each other’s software these relationships become pair-wise! The implication of that is that the marginal increase in software ecosystem leverage and abstraction (the vertical dimension and to my mind software beauty) we can gain with a given input of money (what a teacher of mine refereed to as “green energy”) declines explosively with the number of companies involved. In other words, where L = lawyers and B = given level of abstraction, and N = # entities, L ~= B*N!. [Um.. or probably something like that, I'm too lazy to think it through right now. But trust me, it's big.]

Put even more simply, very few (but > 0) companies are going to want to base their corporate strategy on buying a commercial license to GPL code that they don’t control and that anyone can create a free version of. Very very few people are going to want to make such an arrangement with two partners. And vanishingly small numbers of companies are going to want to do that with more than n partners. We end up with an Ecosystem that looks like this:

What are we left with? Well, the same GPL ecosystem we had before, unless the logic of arguments like this convinces others that a more permissive ecosystem would be a better home. But the commercial situation is actually much worse. Again, now, we can’t even use a dependency’s API without a license! In reality what we would end up with is a separate commercial ecosystem (Microsoft..) and a separate copyleft ecosystem (Gnu..) and a set of separate mixed ecosystems either dominated at the top by small cartels benevolently bending the GPL rules (Linux, Redhat, ??) or controlled by single entities with questionable agendas (MySQL, Oracle).

Which world do you want to live in? Is there a way to live in both? Perhaps self-evolving, decentralized, dynamic software development and deployment systems will convert the distinctions between these various models into not much more than random noise. I’ll return to the diagram from my previous post with an invitation to think about what sorts of software swans might emerge from the networks built by ugly ducklings like OSGi and P2 Aggregator.

Update 11/11 responding to comments: I'll take poster's privilege here and respond to the very stimulating comments below..my responses simply got too long for the comment filter.

I have learned a lot here -- mostly not to make generalizations, even (especially) about a "General" license! So perhaps some cleanup on terms here first..

First Eric, thanks for your kind words. To clarify on the graphs. For each set to try to keep things simple, I'm only drawing the common cases. For instance, in the commercial case Company 0 can license back from Company 1 as well, but I've left that off. Is that what you meant?

After the helpful clarifications from Simon and Laurent, perhaps we can simplify things along a couple of dimensions.

1. I am referring to the "hard" GPL only. A couple of people brought up LGPL in the context of an overall GPL based ecosystem. To be clear I am referring to the GPL sans linking exception and I think that conflating GPL with the LGPL or GPL + L is literally semantics in the sense that while there is a cultural and labeling linkage between the various flavors the implications of each are completely different. LGPL and EPL are even somewhat compatible. To that extent, they don't make an ecosystem per se but only per definition/branding. So for LGPL or GPL + linking you could just use the same graph as for Commercial + Weak-Copy left system. There is the added benefit that you can mix the strong and weak copy-left components more easily so perhaps there should be a fifth set of graphs in there. That's very interesting politically, but let's leave that aside for now. I'll just say that I love Simon's comment "In fact all would be easier if the EPL had been the LGPL". How true that is!

But the main point is that to the extent a developer chooses to use pure, unadulterated GPL, the last scenario applies. I note that in practice this is how we really think of it -- as separate beasties. If someone says "we use LGPL" then people thinking about commercializing will say "OK, that's kind of a pain in the ass, but we can probably live with it".

I am concerned with the non-ideological authors only. By this I mean, to use a very loaded term, the "exploitive" GPL users who use GPL to meet their own business goals in a protective way -- not because they want software to be free but because they want to charge for it. And so my argument as it were is with sub-groups of the two groups that Simon mentions: [groups] who ..accept to sell a "proprietary" version and those who prevent competitors from dual licensing stuff based on their software to clients who did not buy their proprietary license first. Now, I submit that there are a lot of companies and organizations that meet this definition.

I appreciate Simon's comment that "But everyone knows these players, and no one trusts them to build software that we want to dual-license later on" and it sounds like the honest brokers make middle-ware exceptions and so on. I don't have enough experience with GPL licensing world to know how well that works, and it is gratifying to hear it. But still, there seem to be a lot of machinations there and various combinations and I imagine it can be a bit of a minefield. I can't help but think that it would be simpler if people just settled on weak copy-left for the majority of cases, perhaps keeping some of the real crown jewels aside.

There is of course much gray in this area. The most annoying and unexpected and all too common in my experience are with University sponsored projects. I imagine a typical scenario is that somewhere along the way someone tried to convince the University administration to go open source and in order to make that argument they say "if we use GPL, we can always make money licensing it commercially to corporations". Of course, in most cases that never really happens but the damage (from my perspective :) ) is done. On reflection, it I guess that means that it is ironically the gray area that I'm concerned about. Those companies and institutions that have good intentions but that have a bit more of an "I'm hungry now!" attitude or more ideological resonance with strong GPL. These are generally the places that are making really cool pieces of code that would be really useful to a lot of people, and that a lot of people use, but that then can't be integrated with the rest of the commercially oriented eco-system. This is where I think there is a tremendous amount of potential lost and I'd like to find ways to make those worlds work or grow together better.

I am not making an argument against the ideology of GPL, but against it's suitability as an ecosystem and the corresponding "have your cake and eat it too mentality". So, my answer to the rhetorical question of the title is "yes". :) I really do have affection, ambivalent affection, for the pure GPL approach as an exercise in economic and cultural hacking. Are we all seeing the same thing then when as Laurent says, "[GPL has] goals that are beyond 'building a healthy ecosystem'" and Simon says "dependency graphs miss because the GPL is not as much about developing a thriving ecosystem as about guaranteeing freedom in the long run."?

That's fair, and I just want people to see that. I'm just arguing that GPL + commercial is not as healthy weak copy-left + commercial or that they are even close, at least in the short to medium term. In other-words, my argument is aimed at those companies or individuals who want to make money off of their software that they should choose weak copy-left over GPL. Maybe it's obvious that strong GPL and commercial software don't really mix as well as some people claim and all of these graphs are just making an obvious point more complicated than it needs to be! People who are making the decision to participate in GPL should see it as a long-term investment in freedom and that isn't compatible with how it is being used in many cases. Perhaps it is dual-licensing itself that should be prohibited under GPL! And yes, I realize that that would be a hard thing to accomplish legally.

Let's see, this response is way too long already. I think most of Simon's arguments re: Java are spot on. But again here I'd say that the reason GPL worked is not because of its own goals or ideology but because its ideology happens to match up well to narrow corporate interests. Strange bedfellows..

Why have you not covered the possibility to dual license under GPL *and* EPL? This scenario is the proper way to incorporation of GPL content in the ecosystem you are describing.

To Laurent's question "Why have you not covered the possibility to dual license under GPL *and* EPL? This scenario is the proper way to incorporation of GPL content in the ecosystem you are describing." Yes, that is what I would like everyone to do. :) But note that the people who are in the categories I describe in (2) will not do that, because doing so get's rid of all of the perceived advantages. In other words you can lump that into the weak copyleft + commercial category because that is where it goes. By the way for anyone who wants a GPL license for my EPL code, let me know and I'll be happy to grant it! I'll give you a commercial license too if you feel like paying me for it.

Now I'd like to ask for some advice from the GPL experts, since I think people have said in response to this and the previous post that my hack isn't really at all a hack. Actually, I don't think so either, I just meant it in the most general sense of a useful but somewhat ugly way around a particular engineering issue in a way that might not have been contemplated by the original authors. Does that mean that it would be perfectly ok to use GPL code in the way I describe, without even using the P2 aggregator bit? If one simply created a generic API, wrote a plugin for GPL code -- of course while providing all of the code for that plugin under GPL -- and then simply bundled that plugin and it's GPL code with the software itself that that would be ok?

Can The Dynamic Duo Free Software from the GPL?

noreply@blogger.com (Miles Parker) — Tue, 09 Nov 2010 19:45:00 +0000

Disclaimer: Hey I'm not a lawyer, ok? Get your own damn legal advice. In fact, if anything I say makes sense to you, then I wish you'd explain it to me, because it sure doesn't make any sense to me.

Updated 11/10: To understand my motivation for this post, you might want to read my next post, which explains why I think GPL is not the best choice for building healthy software ecosystems.

Updated 11/9: I got some feedback on this blog which -- fairly I think -- took me to task for being too harsh on GPL and FSF. My intention was to express some frustration, poke a little fun at some sacred cows, and offer a creative alternative to the dilemma that so many of us face all of the time when we have to choose between more permissive flavors of software licenses like BSD and EPL, and the much more restrictive GPL license.

I thought I was clear about this, but if there is any confusion, I am not attacking people who make the choice to use GPL. I've considered using the GPL for projects in the past, and I might again in the future. For what it's worth, while I would like to use the software I mentioned below -- and dozens of others -- I'm just fine with the fact that I can't. And I'm embarrassed contemplating the image I may have created of "ranting" against the injustice of the GPL license and laying awake at night plotting ways around it. After all, in the end, I can simply recreate that functionality. The next iteration is always able to learn and grow from the last, and providing software under a license that others can use freely will always give such software a leg up. Anyway, let me assure everyone that personally this really isn't a life and death issue.

But on the other hand, as a software developer, from an almost aesthetic point of view, there is nothing that frustrates me more than seeing the wasted energy spent on having people work on the same functionality for multiple projects under multiple licenses and organizations, when those people could be working together and getting much more accomplished with higher quality. (Yes, I do see the benefit of competition and diversity, and I think there is a value in that, but ultimately the gains from that are overwhelmed by the losses to cooperation.)

The central point of my IP engineering straw-man was to get people to think about alternatives. While I don't see anything wrong with people choosing a particular license, I don't see anything wrong with creative engineering around that license either. We do it all the time. Why should people have to go to the extreme of re-writing things so that everything goes through a socket when there might be better alternatives just to satisfy an attempt by GPL to plug holes that are ultimately un-pluggable? My view is that the terms of the GPL just aren't aligned with the distributed, dynamic way that people are doing software now, and this contradiction is going to only become more apparent over time. If I can help that process along, I'm proud to.

And I did intend to prod FSF a bit, because frankly I think they could use the prodding. There isn't something fundamentally bad with their approach. As I say below, I think there is a lot to admire about copy-left -- it's a daring, ambitious, creative and fascinating response to a lot of issues in our society that go far beyond software licensing. But as in most cases, ambition married to dogmatism leads in the end to contradiction and ends up requiring reliance on half-truths and the expression of naked coercion to support itself. And so yes, I do think that the claims on the FSF website are absurd, disingenuous and actually harmful to the ultimate aim of the FSF, if that aim is education and not indoctrination.

But no, I didn't mean to do it in a nasty way. My standard is to not say something online that I wouldn't be willing to say to someone's face. In an imaginary barstool conversation with someone from the FSF I'm afraid to say that I could imagine myself saying "you know on your website, the part where you say... that's total bullshit, isn't it?" I have had those kind of conversations in a friendly way in many contexts -- and taken as good as I gave -- but perhaps that's the kind of conversation that requires a few beers to go along with it. Anyway, I'm leaving the original post as is, because I don't want to alter history, but with an offer of free beer to my GPL friends when I see you non-virtually.

Recently I wanted to use some really nifty software within my own projects. I'm not going to name the particular project here just because a) it isn't relevant to the argument, b) my concern is not restricted to any one project, and c) I'm also an independent software developer, and respect the struggle that other small companies have trying to find the right strategy. The important point is that there is great potential for collaboration with said software but it uses GPL, which means I can't use it because the the two licenses are fundamentally incompatible.

The software also happens to be based on EPL code -- which I guess makes it technically incompatible with itself! I can't help but say that this approach seems unfair to me. Basically, one is taking the attitude that "I'll use other developer's software under their more open terms, but then release my software under terms that prevent those developers from doing the same". That's fine for commercial software, but in my opinion not so good for "open" software as it makes the relationship one-way. That's a subject I hope to return to in a later post, but ultimately, the banker's question has to be "why participate in a collaboration where the GPL entity is the only company able to make money off of the collaboration?"

As I researched the whole issue and went to others for ideas and advice, I headed over to the GPL site, and found myself looking at pitch perfect double-speak:

So now, let's see... If I want to use GPL Software X, and I decide to include that in my EPL or commercial release, then the Free Software Foundation will use the full power of the courts to prevent me from doing so. I can't buy anyone a beer and I can't even write what I want to. What sort of "freedom" was that again? In what sense exactly am I not "restricted"? I think what the page is really trying to say is "Freedom is Restriction". Hey, I understand and am (ambivalently) admiring of the whole idea of GPL, but could we drop the hypocrisy and variations on a theme by Orwell? Please?

Anyway, I guess I'm not going to get what I "deserve" out of the GPL, so is there a way to liberate it instead? After thinking about it for some time it dawned on me that there were some tools in the Eclipse jail-breaking kit that just might fit the bill.

The first thing we need to understand is what won't work. I can't have a direct dependency on the GPL code, obviously. I can't build something that requires the copyrighted GPL software to compile, and I obviously can't deliver it either.

But, what if I encapsulated the functionality that I wanted into an independently developed set of APIs? Then, I could create a separate code base for a plugin that implements that generic API and delegates the calls to the target GPL API. (Note that I can release the same or other software under various licenses. For example, I could make my Generic API available under EPL and/or BSD.) BAM!!

Perhaps we can escape this diabolic trap after all. We can see in the following diagram that we've zig-zagged around any dependency issues. The OSGi plugin loads the GPL functionality at runtime, not design or compile time. Take that, fiendish enslavers of software! SOCK!!

But there is still a problem here. Though no commercial software vendor would ever claim such a right and get away with it, the FSF actually wants to intrude on what software a program is allowed to link to dynamically. For some reason I can't fully articulate, as a lover of software the idea that someone could prevent you from executing something from within a program after the software is written -- even though you have the rights to use both pieces of software independently -- seems absurd to me, and even a bit wicked.

Now, as far as I know, the FSF has never had any success convincing a court of the enforceability of this unique approach to deconstructing copyright. It essentially requires a form of copyright protection that can see into the future and read minds. ("I just know that he wrote that software with the intention of allowing other people to link it to my software.") But in any case, the only hope of actually enforcing this lunacy rests on software delivery, not development. That's because each software component isn't at all related until the whole distribution is assembled, packaged and provisioned.

That is an extremely important point, because it means that as say an EPL developer, I would be free to use the above strategy without putting myself or my software under any jeopardy at all. (Now would be a good time to scroll back up to the top and read that disclaimer again..) I could even put that software on a server somewhere and let people build it themselves. (OK, Richard Stallman might have something to say about that, but in that case just put your hands over your ears and repeat "nanananan..I'm not listening...I'm not listening..." Or maybe a freeze ray would be in order.) The only trouble -- again, under the most ludicrous potential court ruling, but it isn't like ludicrous rulings never happen -- would be when I actually tried to deliver software to someone. If I bundled said GPL software with my project or product, and I hosted that bundle on my site, I maybe sort of possibly could be violating the GPL license. Somehow.

So you may have been wondering how P2 Aggregator fits into the picture. Here's how: nobody ever actually delivers the software at all! Instead, the end user selects the components that he or she wants from an aggregator site. If you think about it, aggregator is really a dynamic plugin mechanism, but for deployment rather than run-time. And, like plugins, it allows us to insert indirection into the dependency chain. You can assemble stacks of dependencies in any arbitrary way as long as those dependencies are satisfied. So, let's use dynamic deployment in the same spirit we used dynamic development. POW!!

Dependencies are ultimately specified in a file somewhere but in such a way that the software as delivered meets the appropriate license for the appropriate usage path. In other words, the GPL site "conveys" the GPL content, including as appropriate the dual-licensed proxy-API and plugin code. The EPL licensed project provides all of its code and thus of course avoids any potential issues of license incompatibility. And the DMZ site provides the generic API (and optionally the proxy plugins) under BSD license, which is compatible with both. WHAP!!!

The really subversive part is that a commercial vendor might even be able to use this method to deliver software using GPL functionality. KA-POW!!!!

Note that the P2 aggregate approach is very different from the kind of "aggregate" discussed in the GPL FAQ. In a typical aggregate like a Linux DVD Distro (does anyone actually make those anymore?), all of the software is put in one place. Instead, with a P2 Aggregator provisioned application, the customer assembles the software for his or herself, just as he or she does when downloading applications from the web. The GPL EPL software dependencies never exist anywhere else! So there is simply no sense in which the two sets of software could be said to incorporate each other.

As a thought experiment, imagine a customer who wants to use a mixture of open source and commercial software to edit photos. If she installs PhotoShop and a GPL'd Photoshop plugin, she couldn't be accused of violating copyright, could she? What if she went to an independent website and that website had a link for downloading Photoshop and another one for downloading the GPL plugin? Would that website be violating the GPL copyright? What if that website were instead hosted by Photoshop. Would that magically cause Adobe to be in violation of copyright? Now, imagine that both Photoshop and the plugin were hosted at oh, say the Mac App Store? Would the App Store be in violation? What if instead that user went to an Eclipse Market provider and selected a commercially licensed Eclipse image editor, together with a GPL-based plugin, which were then installed together? As far as I can see, the distinctions all quickly reduce into absurdity.

I'd love to hear your point of view whatever it is. To avoid wasting a lot of time, I'd especially like to hear from you if you think my idea won't work! (And in the interests of giving the Eclipse Foundation legal team a sound night's sleep, no, I am definitely not planning to test this theory out on an Eclipse hosted project.)

Well, kids, can Captain Aggregator and The Plugin free captive software from Dr. GPL? Or is our attempt to free ourselves from the forces of [...hmm... "forces of people with a different point of view" doesn't really have much of a ring to it, does it] doomed to failure? Quite a cliff-hanger, eh?